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PREFACE 


This  technical  paper  documents  an  Initiative  by  the  Air  Force  Hu>an  Resources 
Laboratory  to  conduct  basic  research  on  learning  that  could  be  used  In  the  training  of 
Air  Force  personnel.  The  technical  direction  for  the  Initiative  was  provided  by  Lt  Col 
David  L.  Payne,  who  Inspired  the  efforts  that  produced  this  document.  Dr.  Robert  M. 
Gagnd  was  also  a  very  significant  contributor.  During  most  of  this  effort,  he  was 
serving  at  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  as  an  expert  consultant 
while  on  leave  from  Florida  State  University.  He  graciously  continued  to  contribute  to 
the  effort  after  returning  to  the  University. 

This  effort  represents  a  portion  of  the  technical  program  of  the  Manpower  and 
Personnel  Division  of  the  Air  Force  Human  Resources  Laboratory  (AFHRL/MO).  This  effort 
also  was  pertinent  to  the  tect  .leal  program  of  the  AFHRL  Logistics  and  Human  Factors 
Division  (AFHRL/LR) .  This  effort  was  performed  as  Task  09,  Workshop  on  Instructional 
Technology,  under  Contract  F3361 5-82-C-0006  (SB5448-82-C-0099)  with  Universal  Energy 
Systems,  Inc.  This  contract  1$  docueented  under  Work  Unit  1710-00-08,  Contributlve 
Research  In  Logistics  Factors  and  Training  Techniques,  with  Mr.  Craig  Casslno  serving  as 
Work  Unit  Monitor.  Mr.  Edward  Boyle  was  the  technical  representative  of  AFHRL/LR  on 
this  effort.  For  this  effort,  Drs.  Goldstein,  Glaser,  Royer,  and  Shuell  served  as 
employees  of  Universal  Energy  Systems,  Inc.,  under  the  general  supervision  of  Dr.  Ross 
L.  Morgan.  Dr.  Goldstein  served  as  Chairman  of  the  group.  In  addition  to  authoring  two 
chapters  of  the  present  report.  Dr.  Goldstein  coordinated  the  efforts  of  all  of  the 
authors. 

Significant  technical  contributions  to  this  effort  were  made  by  Individuals  who  met 
with  the  authors  and  briefed  them  on  relevant  research  and  Issues.  These  Individuals 
Include  the  following:  Dr.  Raymond  E.  Chrlstal  and  Dr.  Hendrick  w.  Ruck  of  AFHRL/MO; 
Dr.  Joseph  Y.  Yasutake  of  the  AFHRL  Training  Systems  Division;  Dr.  William  E.  Montague 
of  the  Navy  Personnel  Research  and  Development  Center;  and  Dr.  Joseph  S.  Ward  of  the  US 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences.  These  Individuals  also 
reviewed  drafts  of  the  present  document,  as  did  Dr.  William  C.  Tlrre,  AFHRL/MO,  who 
provided  substantive  review  of  each  paper.  In  addition  to  those  whose  contributions 
have  been  acknowledged,  many  unnamed  Individuals  served  comnendably  on  the  team  that 
produced  this  technical  paper.  It  Is  hoped  that  appreciation  for  their  services  has 
been  acknowledged  appropriately  at  other  times  and  places. 
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LEARNING  RESEARCH  LABORATORY:  PROPOSED  RESEARCH  ISSUES 


CHAPTER  1 
INTRODUCTION 

The  Air  Force  Human  Resources  Laboratory  (AFHRL),  Is  preparing  plans  for  a  learning  research 
laboratory  that  can  accomplish  generallzable  research  on  problems  of  training  common  to  many  Air 
Force  jobs.  The  purpose  of  such  a  laboratory  would  be  to  conduct  basic  research,  in  learning  and 
training,  that  would  have  implications  for  the  training  functions  of  the  Air  Force. 

To  help  formulate  the  technical  program  of  a  learning  research  laboratory,  several  noted 
researchers  were  asked  to  review  the  pertinent  needs  of  the  Air  Force,  compare  those  needs  with 
the  relevant  technology,  and  propose  approaches  to  a  productive  research  program.  As  a  first 
step  toward  the  basic  objective,  a  workshop  was  conducted.  This  workshop  was  attended  by  the 
authors  of  the  present  document  and  representatives  of  AFHRL,  the  Navy  Personnel  Research  and 
Development  Center  (NPRDC) ,  and  the  Amy  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI).  During  the  workshop,  representatives  of  AFHRL  presented  Information  about  the 
Air  Force  training  process  and  related  research  programs  in  AFHRL.  The  representatives  of  NPRDC 
and  ARI  provided  summaries  of  their  research  programs  and  offered  useful  advice  on  further 
research. 

Following  these  presentations,  the  authors  presented  preliminary  versions  of  their  papers. 
Their  presentations  had  the  following  objectives: 

1.  State  the  author's  view  of  contemporary  research  and  theory  on  cognitive  learning  and 
individual  differences  in  cognitive  processes  that  are  relevant  to  the  planning  of  the  laboratory 
and  its  program. 

2.  Identify  the  research  problems  in  the  area  of  basic  learning  and  training  research  that 
are  considered  to  be  promising  in  terms  of  valuable  outcomes. 

3.  Indicate  the  applicability  of  the  research  areas  to  problems  of  Air  Force  training. 

Each  of  the  papers  was  followed  by  discussions  among  all  of  the  participants.  Following  the 
workshop,  the  authors  developed  formal  papers.  Those  papers  were  circulated  for  comments  to  all 
individuals  who  attended  the  workshop.  The  conments  also  were  circulated  to  all  participants. 
After  this  process,  revised  papers  were  developed  and  are  presented  here.  The  final  chapter 
provides  a  summary  of  the  research  issues  that  are  discussed  In  the  various  chapters. 

The  participants  in  this  effort  hope  that  the  research  ideas  contained  in  these  papers  will 
contribute  to  a  productive  research  agenda  for  an  Air  Force  learning  research  laboratory. 


CHAPTER  2 


GENERAL  ORIENTATION  TO  ESTABLISHMENT  OF 
A  LEARNING/TRAINING  RESEARCH  LABORATORY 

David  L.  Payne 

Air  Force  Human  Resources  Laboratory 

Robert  M.  Gagr>£ 

Florida  State  University 

The  Air  Force  has  223  enlisted  specialties  In  48  career  fields  and  has  218  officer 
specialties.  Training  Is  conducted  by  the  Air  Training  Command  (ATC).  For  enlisted  airmen,  a 
period  of  6  weeks  of  basic  training  Is  usually  followed  by  assignment  to  a  school  where  they  are 
taught  the  knowledge  and  skills  relevant  to  a  particular  type  of  job  assignment.  Schools  teach 
courses  In  each  of  the  broad  career  fields,  such  as  aircraft  mechanic,  connunl cations  systems 
maintenance,  personnel  specialist,  and  accounting  and  finance.  In  addition  to  resident  training, 
operational  field  training  and  on-the-job  training  (OUT)  may  be  given  In  order  to  refresh  skills 
Infrequently  used,  or  to  meet  new  requirements  because  of  the  Introduction  of  system  changes. 


The  Need  for  a  Learning  Research  Laboratory 


The  need  for  a  laboratory  to  Investigate  problems  of  learning  arises  due  to  problems 
associated  with  conducting  certain  types  of  research  In  operational  training  environments.  In 
the  Air  Force,  resident  training  courses  are  highly  structured  to  present  Information  required  to 
bring  the  student's  knowledge  up  to  predetermined  levels  of  acceptability.  The  courses  are 
designed  to  do  this  In  the  most  efficient  manner  possible,  with  production  quotas  dictated  by  the 
needs  of  the  Air  Force.  These  operational  requirements  often  make  It  extremely  difficult  to 
conduct  research  which  requires  manipulations  In  methods  of  Instruction,  course  length,  and 
related  factors  of  interest.  In  this  setting,  we  often  must  forsake  certain  requirements  of 
appropriate  research  design  (for  example,  use  of  control  groups)  to  avoid  disruptions  in  the 
ongoing  training  system.  Also,  changes  In  training  materials,  curricula,  media  and  methodologies 
are  Introduced  Into  the  operational  setting  on  a  periodic  basis.  These  changes  are  often  not 
properly  evaluated  In  an  experimental  setting  prior  to  Implementation,  due  to  constraints  In  the 
operational  environment. 

This  Is  not  to  say  that  the  Air  Force  Is  unwilling  to  disrupt  their  day-to-day  training 
regimen  If  the  research  holds  great  promise  of  operational  benefits;  Indeed,  this  has  happened  In 
previous  studies  In  pilot  selection,  computer-assisted  Instruction,  and  other  areas.  However, 
demonstrating  “great  promise*  on  an  a  priori  basis  for  training  research  Is  often  not  possible 
for  basic  research  efforts.  As  part  of  the  AFHRL  learning  abilities  research  to  be  described  by 
Dr.  Chrlstal,  we  will  Identify  Individual  differences  In  abilities  that  will  generate  hypotheses 
to  be  tested  by  research  In  technical  training.  However,  It  may  well  be  difficult  to  convince 
the  operational  trainers  that  the  test  of  a  theory  Is  worth  the  disruption  of  ongoing  training 
activities,  the  cost  of  developing  two  courses,  and  so  on.  A  research  laboratory,  separate  from 
normal  training  operations,  would  provide  the  opportunity  to  conduct  limited  tests  in  a  far  less 
disruptive  and  less  costly  manner. 


Areas  of  Research 


What  kinds  of  research  on  learning  and  Instruction  need  to  be  included  In  the  program  we  are 
planning? 
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I'm  sure  you  don't  expect  me  to  provide  any  specific  answers  to  that  question,  because  that's 
what  we're  asking  you  to  give  us  In  the  presentations  you  make  and  the  discussions  that  grow  out 
of  them.  Here,  though,  are  some  very  general  areas  that  appear  to  contain  the  kinds  of  problems 
faced  by  Air  Force  training: 

1.  Studies  need  to  be  conducted  pertaining  to  the  acquisition,  retention,  and  reacquisition 
of  Air  Force  Job-relevant  skills  and  knowledges.  Without  highly  controlled  experimental 
conditions,  acquisition  rates  are  extremely  difficult  to  measure.  The  Air  Force  now  trains  to 
certain  percentage  standards  of  performance,  with  no  examination  of  the  effects  of  individual 
differences  In  achieving  these  levels  of  performance.  One  of  the  problems  In  forecasting 
knowledge  and  skill  acquisition  rates  Is  that  there  may  be  Interactions  between  aptitude  and 
training  methods.  For  example,  some  Individuals  may  learn  faster  from  auditory  presentations 
than  from  visual  presentation  of  training  materials.  Other  questions  of  Interest  In  this  domain 
Include  the  following:  Can  long-term  retention  be  predicted  from  Indicators  of  short-term 
learning  or  of  Immediate  recall?  If  transfer  depends  on  similarity  or  on  identity  of  elements, 
what  are  the  elements?  Is  the  schema  a  useful  way  to  conceptualize  the  elements  that  need  to  be 
similar?  What  are  the  effects  of  continued  practice,  and  what  Is  accomplished  when  a  skill  Is 
automatl zed? 

2.  There  have  been  various  studies  Indicating  a  relationship  between  aptitude  and  training 
time.  However,  at  the  present  time,  no  one  can  express  aptitude-training  time  tradeoffs  with  any 
degree  of  precision.  If  one  were  to  raise  the  aptitude  level  of  Individual  groups  assigned  to  a 
training  course  by  X  points,  what  savings  In  training  time  could  be  realized  in  bringing 
personnel  to  the  same  level  of  competence?  If  the  quality  of  recruits  decreases,  as  Indicated  by 
lower  aptitude  scores,  how  much  additional  training  time  will  be  required  to  achieve  the  same 
level  of  competence?  Are  there  some  courses  In  which  an  acceptable  level  of  competence  cannot  be 
attained  with  lower-aptitude  accessions?  Questions  of  this  sort  suggest  a  need  to  undertake  some 
fundamental  Investigations  of  "time  to  learn"  and  how  It  Is  affected  by  the  nature  of  the  skill 
being  trained  and  by  the  aptitude  of  the  learner. 

3.  A  critical  need  exists  to  provide  an  appropriate  research  setting  In  which  job 
performance  criteria  can  be  used  to  validate  current  and  proposed  ability  tests.  Although  most 
managers  accept  that  our  current  selection  tests  are  reasonably  valid  for  predicting  school 
success,  they  often  ask  the  question,  "How  well  do  the  tests  predict  performance  on  the  job?" 
This  Is  a  fair  question  which  the  research  community  has  not  adequately  answered,  primarily  due 
to  technical  problems  associated  with  using  field  job  performance  as  a  validation  criterion. 
These  problems  have  proven  so  severe  that  many  of  us  doubt  they  will  ever  be  overcome  without 
major  changes  to  our  current  validation  strategies.  I  suggest  that  a  primary  objective  of  the 
proposed  research  laboratory  should  be  to  collect  reliable  performance  data  for  test  validation 
purposes,  thus  providing  reliable  "Intermediate  criteria." 


Representative  Tasks 

In  order  to  have  a  good  basis  for  the  research  in  such  a  laboratory,  we  Intend  to  select  and 
use  some  reference  tasks  representative  of  what  people  must  learn  to  do  in  Air  Force  jobs.  Dr. 
GagnA  has  spent  some  time  searching  for  components  of  some  Air  Force  enlisted  specialties  that 
could  serve  as  reference  tasks,  to  be  learned  over  a  time  span  of  several  days  or  less.  He  Is 
trying  to  find  the  kinds  of  job  components  that  would  require  dealing  with  the  following  kinds  of 
things:  (a)  maintaining  electrical  systems;  (b)  maintaining  mechnlcal  systems;  (c)  computing 
with  numbers;  (d)  composing  written  messages;  and  (e)  sorting,  classifying,  and  coding 
Information  on  forms. 


The  Idea  would  be  to  select  some  convenient  job  components  from  these  specialties  or  from 
others  like  them.  These  components  (not  necessarily  single  tasks)  would  be  taught  to  airman 
recruits  over  a  reasonably  short  period  of  time  and  under  laboratory  control  of  conditions. 
These  would  then  become  the  reference  tasks  to  serve  as  criteria  for  performance  effects  of 
Instructional  variables  In  experimental  studies. 


It  Is  often  a  difficult  part  of  learning  research  to  select  learning  tasks  that  represent 
"the  real  world."  Investigators  often  have  to  think  hard  to  devise  tasks  that  will  simulate 
problems  or  other  learning  situations  that  are  realistic.  Our  approach  Is  to  try  to  select  and 
present  pieces  of  actual  jobs.  This  seems  to  us  a  part  of  the  answer  of  how  to  begin.  Other 
suggestions,  we  hope,  will  be  coming  from  this  workshop. 


CHAPTER  3 


RESEARCH  ON  LEARNING  ANO  RETAINING  SKILLS 


Robert  M.  Gagne 
Florida  State  University 


Having  In  Bind  the  need  for  research  that  will  shed  light  on  the  conditions  of  skill  learning 
and  skill  Maintenance,  as  It  Might  be  conducted  In  an  Air  Force  laboratory,  I  have  prepared  the 
following  account  of  what  seeM$  to  Me  to  be  proMlsIng  lines  and  directions  for  a  research 
program.  This  begins  with  an  account  of  a  set  of  "benchmark"  learning  tasks,  derived  directly 
froM  Air  Force  jobs,  and  Intended  to  serve  as  representatl ves  of  those  jobs.  Following  this  are 
descriptions  of  lines  of  research  on  skill  learning.  Mental  effort  as  an  Indicator  of  task 
difficulty,  skill  retention,  and  relearning.  Description  of  each  of  these  major  areas  Is 
accompanied  by  a  list  of  references,  collected  by  area,  at  the  end  of  this  paper. 


BenchMark  Tasks  for  Learning  Research 


Frequently,  tasks  chosen  for  studies  of  learning  In  Military  settings  cover  a  range  from 
fairly  slMple,  real  tasks  to  more  complex  tasks  that  are  designed  to  simulate  actual  military 
tasks,  and  Include  learning  tasks  designed  to  Incorporate  the  cognitive  processes  of  actual 
tasks.  An  example  of  a  comprehensive  study  of  this  sort  Is  Fox,  Taylor,  and  Cay  lor  (1969). 
Tasks  used  by  these  Investigators  Included  the  actual  task  of  rifle  assembly  and  disassembly  and 
a  simulated  task,  called  "missile  preparation,"  that  required  the  learning  of  a  34-step  procedure 
In  responding  to  controls  on  a  panel.  In  more  recent  Army  work,  learning  and  retention  of  a 
number  of  actual  job-tasks  have  been  studied  In  field-type  research.  For  example,  Goldberg, 
Orll lings,  and  Dressel  (1981)  used  the  actual  task  of  boresighting  and  zeroing  the  M60-A1  main 
gun.  Hagman  (1980)  Investigated  several  parts  of  the  job  of  charging  system  output  using  the 
500A  Sun  Test  Stand  generator;  and  Holmgren,  Hllllgoss,  Swezey,  and  Eaklns  (1979)  looked  at  a 
variety  of  tasks  oriented  around  the  machine  gun,  the  squad  radio,  tube  artillery,  and  radar 
equipment. 


Selection  of  Job  Components 


There  are,  then,  different  approaches  to  the  selection  of  representative  job-tasks,  which  are 
therefore  valid  In  the  sense  of  possessing  verisimilitude  as  judged  by  observers  who  are  informed 
and  unbiased.  Obviously  the  most  direct  method  Is  to  select  a  group  of  tasks  that  are  Integrally 
related  to  some  coherent  job  operation.  This  organized  set  of  tasks  may  be  called  a  job 
component.  Sometimes  a  job  component  of  this  sort  may  be  revealed  by  the  organized  set  of  tasks 
called  a  "duty."  But  this  is  not  always  the  case,  and  direct  observation  of  job  Incumbents  may 
be  necessary  to  Identify  appropriate  job  components. 

Once  selected,  the  representativeness  of  a  job  component  may  be  assessed  by  means  of  various 
measures  of  Its  tasks  available  through  the  Comprehensive  Occupational  Data  Analysis  Programs 
( CtDAP )  system.  These  Include  the  percentage  of  personnel  performing  the  tasks  and  the  rated 
training  emphasis.  The  job  component  selected  should  contain,  as  a  minimum,  tasks  which  have  a 
high  percentage  performance  by  Incumbents  of  the  Air  Force  Specialty  (AFS) .  Otherwise,  it  seems 
appropriate  that  the  job  components  represent  a  range  of  different  AFSs,  in  the  following  manner: 
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1.  A  procedure  of  10  or  more  steps  outlined  In  a  technical  manual  pertaining  to  checking  the 
operation  of  complex  equipment  (possibility:  AFSC  423X5,  Aerospace  Ground  Equipment  (AGE) 
Mechanic). 

2.  A  procedure  of  six  or  seven  steps  to  be  recalled  without  the  aid  of  printed  text. 

Involving  removing  or  Installing  mechanical  components  (possibility:  AFSC  423X5,  AGE  Mechanic). 

3.  The  Inspection  and  calibration  of  a  test  set,  requiring  responses  to  a  number  of 

different  dials  and  Indicators  (10  to  16)  on  a  panel  (possibility:  423X0,  Aircraft  Electrical 
Systems  Specialist). 

4.  The  composition  of  relatively  brief,  partially  structured  messages  In  memoranda  and 

correspondence  (possibility:  732X0,  Personnel  Specialist). 

5.  The  carrying  out  of  "paper"  procedures,  such  as  disbursement  and  accounting 
(possibility:  672X2,  Financial  Service). 

6.  The  computation  of  trends  and  other  numerical  measures  (possibility:  691X0,  Cost  and 

Management  Analysis  Specialist). 


Validating  Aptitude  Tests  and  Batteries 

The  representative  nature  of  job  components  means  that  they  can  play  the  role  of  Intermediate 
criteria  against  which  aptitude  tests  can  be  validated.  This  potential  use  serves  to  confirm  the 
desirability  of  a  suitable  range  of  job  components  In  different  aptitude  areas.  Referring  to  the 
previously  described  possibilities,  the  Electronics  area  Is  represented  by  (1)  and  (3), 
Mechanical  Is  represented  by  (2),  General  Is  represented  by  (4)  and  (6),  and  Administrative  Is 
represented  by  (5).  In  view  of  the  current  difficulties  and  suspected  Inappropriateness  of 
school  grades  as  criteria  for  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  and  other  tests, 
this  particular  use  of  job  components  In  a  laboratory  setting  may  be  seen  as  a  requirement  with 
high  priority.  Correlations  would  be  obtained  between  job  component  performance  scores  and  the 
various  sub-scores  of  ASVAB,  as  well  as  with  individual  tests  of  aptitude. 


Validating  Job  Difficulty  Measures 

C00AP  data  Include  ratings  of  task  difficulty  that  are  obtained  by  asking  supervisors  "How 
long  does  It  take  to  learn  the  task?"  This  measure  needs  to  be  validated  by  comparing  It  with  a 
direct  measure  of  "time  to  learn"  on  the  same  task.  Obtaining  the  latter  measure  on  a  job 
component  will  require  some  methodological  development.  Time  to  learn  has  not  been  widely 
employed  as  a  measure,  and  some  arbitrary  criteria  of  learning  and  performance  may  have  to  be  set 
before  meaningful  research  can  be  done.  A  direct  measure  of  time  to  learn  for  job  components 
would  make  possible  interesting  and  valuable  comparisons  with  task  difficulty  as  currently 
assessed. 


Validating  Learning  Measures 

In  a  learning  research  program.  It  may  often  be  convenient  to  present  the  learning  task  to  be 
Investigated  by  means  of  a  computer  with  TV  monitor,  TV  disc,  or  similar  device.  This  means  that 
the  task  will  likely  be  an  abstraction  from  the  "real"  task.  Yet,  such  abstractly  derived  tasks 
may  be  necessary  to  employ  for  the  study  of  such  variables  as  mental  effort,  varieties  of 


practice,  overlearning,  and  automatization.  Sometimes,  research  findings  may  make  possible  more 
or  less  direct  applicability  to  the  planning  of  training.  In  many  Instances,  however,  the 
findings  of  laboratory-type  studies  on  synthetic  tasks  may  be  checked  against  comparable  findings 
with  real  job-tasks,  that  Is,  with  job  components. 


Validating  Cognitive-Skill  Analysis 

With  the  purpose  of  defining  training  requirements,  a  number  of  current  projects  of  the 
field-research  type  propose  to  analyze  tasks  of  Air  Force  jobs.  There  are  two  outcomes  of  this 
kind  of  analysis.  First,  the  analysis  reveals  the  supporting  skills  and  knowledges  of  a  task. 
According  to  Instructional  Systems  Development  (ISO)  procedures,  these  specific  skills  and 
knowledges  are  Incorporated  Into  plans  for  training  and  Into  course  design.  Examination  of  the 
task,  together  with  Its  supporting  skills  and  knowledges,  may  also  yield  a  characterization  of 
the  task  as  partaking  of  one  or  more  critical  cognitive  processes  (such  as  high  mental  effort). 
This  second  kind  of  outcome  of  analysis  Is  likely  to  reveal  some  requirements  for  classification 
testing. 

Supporting  skills  and  knowledges  of  a  task  may  be  validated  by  studies  having  the  framework 
of  transfer  of  training.  Students  who  learn  the  Identified  skills  and  knowledges  should  learn  to 
perform  the  total  task  more  readily  (faster  or  with  fewer  errors)  than  students  who  have  not 
learned  the  same  skills  and  knowledges.  Since  It  Is  extremely  difficult  to  perform  such  studies 
In  the  field,  basic  research  Investigations  that  validate  this  type  of  ’component  analysis’  of 
job-tasks  would  appear  to  be  of  substantial  value.  Of  course,  the  most  Important  validation  Is 
of  the  method  and  not  of  the  specific  skills  and  knowledges  Identified. 

The  relation  of  particular  cognitive  processes  (or  skills)  to  task  performance  can  also  be 
validated  by  correlations  between  basic  cognitive  skills  revealed  by  analysis  and  performance  on 
job  components.  This  Is,  of  course,  simply  a  particular  Instance  of  aptitude  test  validation  as 
previously  described. 


Establishing  Reliable  Measures  of  Performance 

Since  job  components  are  considered  to  play  such  a  crucial  role  In  validation  of  a  variety  of 
testing  and  learning  operations.  It  will  be  a  necessary  goal  to  select  and  devise  measures  of 
performance  that  are  as  reliable  as  possible.  Low  reliability  has  often  been  an  obstacle  to  the 
use  of  job  performance  measures.  Can  reasonably  high  reliability  be  obtained  with  measures  of 
performance  on  job  components? 

The  nature  of  measures  applicable  to  job  components  cannot  be  estimated  ahead  of  time  with 
any  assurance.  Probably,  the  following  characteristics  ought  to  be  aimed  for: 

1.  Criterion-referenced,  performance-based. 

2.  fielding  a  range  of  scores  reflecting  performance  accuracy. 

3.  Having  reliability  that  Is  conceived  as  coefficient  of  equivalence  (Siegel  et  al.,  1979); 
l.e.,  correlation  between  measures  obtained  from  equivalent  Instruments. 


The  requirement  for  objective  measures  of  job  performance  Is  widespread  among  projects 
dealing  with  selection  and  classification  testing,  training  development  and  evaluation,  and  the 
structuring  and  revision  of  AFS  categories.  Many  difficulties  attend  the  attempts  to  collect 
data  on  the  job  performance  of  airmen  assigned  to  various  occupational  specialties.  For  one 
thing,  following  Initial  school  training,  personnel  In  any  single  career  field  become  widely 
scattered  at  operational  bases  In  the  United  States  and  overseas;  the  result  Is  a  thinning  of  the 
ranks  of  airmen  who  could  be  considered  to  constitute  groups  of  comparable  Individuals  for  study 
purposes.  Although  they  may  be  roughly  comparable  at  the  end  of  first-school  training,  airmen  In 
a  single  AFS  encounter  various  job  assignments  and  job  conditions  In  different  commands  and 
different  locations. 


A  second  major  consideration  regarding  the  assessment  of  performance  of  airmen  In  such 
specialties  as  equipment  maintenance  Is  that  It  becomes  an  enormously  complex  task  Involving  real 
equipment,  parts  of  real  equipment,  simulation  equipment,  training  devices,  and  other  materials. 
Because  of  the  necessity  of  employing  these  varieties  of  equipment  and  establishing  standard 
conditions  of  observation,  the  whole  effort  becomes  quite  expensive.  Examples  of  what  may  be 
required  are  contained  In  reports  by  Foley  (1974);  Shrlver  and  Foley  (1974);  and  Shrlver,  Hayes, 
and  Hufhand  (1975). 

It  would  appear  that  widespread  evaluation  of  job  performance  at  a  selection  of  Air  Force 
bases  Is  prohibitively  expensive,  as  well  as  being  disruptive  of  ongoing  operations. 
Accordingly,  It  would  seem  desirable  to  validate  methods  of  job  performance  assessment  In  the 
restricted,  controlled  environment  of  a  laboratory.  Thoroughgoing  procedures  such  as  those 
described  by  Shrlver,  Hayes,  and  Hufhand  (1975)  could  be  established  In  connection  with  a 
relatively  minor  system  or  equipment.  Following  this,  methods  of  simplifying  the  procedures  of 
performance  assessment  could  be  explored  and  tested.  The  aim  would  be  a  set  of  procedures  that 
are  practical  for  on-the-job  performance  assessment  and  that  would  yield  valid  and  reliable 
measures. 


Skill  Learning  Beyond  Initial  Acquisition 


Observations  of  human  performance  that  demonstrates  intellectual  skills  often  raise  questions 
about  the  adequacy  of  the  learning  experience  that  has  produced  those  skills.  Skills  that  are 
not  constantly  used  are  often  forgotten.  Skills  that  are  employed  with  moderate  frequency  may 
have  to  be  subjected  to  a  self-review  process  before  being  used,  rather  than  being  readily 
accessible  for  Immediate  application  to  each  new  specific  situation.  These  circumstances  suggest 
questions  about  the  completeness  or  adequacy  of  learning  and  their  effects  on  skill  maintenance 
and  genera llzablllty. 

Cognitive  psychologists  have  suggested  that  skills  may  progress  through  a  number  of  stages  of 
adequacy  beyond  the  point  of  Initial  acquisition.  Neves  and  Anderson  (1981)  proposed  that 
procedural Izatlon  of  a  skill  begins  with  a  transition  from  knowledge  that  is  declarative 
Information  (evidenced  by  the  learner's  verbal  statement)  to  knowledge  that  is  procedural  In  the 
sense  of  applicability  to  one  or  more  specific  Instances.  Changes  that  go  beyond  initial 
acquisition  of  the  procedure  and  that  may  take  place  either  simultaneously  or  successively 
Include  the  following: 

1.  Composition,  In  which  certain  parts  of  the  total  procedure  are  consolidated  In  such  a  way 
that  “short-circuiting"  of  the  procedure  takes  place. 
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2.  Speed-up,  which  occurs  as  a  result  of  composition  and  when  certain  unessential 
part-activities  drop  out  with  continued  practice. 

3.  Autoaatlclty,  which  makes  possible  parallel  rather  than  successive  processing  of  certain 
activities,  such  as  the  Identification  of  appropriate  conditions  for  employing  a  skill. 
Automlzatlon  results  In  a  performance  that  does  not  Interfere  with  another  task  concurrently 
carried  on  (Shlffrln  i  Dumals,  1981). 

Changes  In  learned  capabilities  which  go  beyond  Initial  acquisition  are  described  by 
Rumelhart  and  Norman  (1978)  as  “tuning. *  As  applied  to  schemata,  the  Idea  of  tuning  may  Include 
changes  that  accomplish  any  of  the  following:  (a)  Improve  accuracy,  (b)  generalize 
applicability,  (c)  specialize  applicability,  and  (d)  determine  the  “default  values,*  or  the 
characteristics  of  the  typical  case.  Generally  speaking,  the  changes  In  capability  described  by 
Neves  and  Anderson  (1981)  are  expected  to  result  from  additional  practice.  The  tuning  changes 
may  also  be  brought  about  with  practice,  but  there  may  be  other  ways  of  effecting  these  changes 
following  the  Initial  attainment  of  the  skill.  The  effects  of  practice  may  serve  to  amplify 
“resistance  to  forgetting*  as  Wlckelgren  (1972)  suggests. 


Research  on  Learning  Beyond  Initial  Acquisition 

Some  areas  of  research  that  appear  to  be  worthy  of  investigation  include  the  following: 

1.  Can  phases  of  learning  beyond  Initial  acquisition  be  Identified,  and  can  the  operations 
for  detecting  them  be  specified?  In  a  procedural  task  of  10  to  12  steps,  learning  could  first  be 
carried  to  the  criterion  of  one  perfect  performance,  and  the  number  of  trials  recorded.  An 
average  number  of  trials  would  be  obtained  for  groups  In  aptitude  categories  1,  2,  and  3.  In  the 
same  categories,  trials  would  be  given  to  represent  150%  and  200%  “overlearning.*  Observations 
would  be  made  to  detect  “composition*  and  “speed-up*  of  parts  of  the  procedure.  For  example, 
steps  that  followed  each  other  by  good  cueing  might  be  expected  to  exhibit  composition  and 
speed-up,  whereas  steps  that  did  not  follow  Inherent  cues  might  not  behave  in  the  same  way. 
Composition  and  speed-up  would  be  detected  by  the  timing  of  part-activities  and  by  other 
methods.  Automatization  would  be  detected  by  the  degree  of  degradation  In  a  secondary  task 
(monitoring  lights,  or  the  like). 


2.  Can  the  phases  of  advanced  skill  compilation,  as  mentioned  in  1,  be  attained  by  methods 
other  than  continued  practice?  As  contrasted  with  “straight  practice,"  alternative  procedures  to 
achieve  composition,  speed-up,  and  automatization  might  be  (a)  specific  training  on 
identification  of  the  cues  leading  from  one  step  to  the  next,  (b)  setting  speed  goals  for  the 
learner,  (c)  practice  of  the  procedure  In  the  presence  of  a  secondary  task  (cf.  Naylor,  Briggs,  A 
Reed,  1968),  or  (d)  various  combinations  of  these  approaches.  In  general,  the  purpose  of  such 
studies  would  be  to  compare  the  effects  of  alternative  methods  with  those  of  straight  practice. 
It  Is  possible  that  such  methods  might  save  time  or  otherwise  accomplish  the  goal  of  skill 
consolidation  more  efficiently  than  would  continued  practice. 

3.  Proceeding  to  Investigate  further  these  advanced  phases  of  learning,  the  questions 
include  the  following: 


a.  Mhat  are  the  effects  of  the  composition  phase  on  retention  of  a  skill,  relearning  of 
the  skill,  and  transfer  of  the  skill  to  related  tasks? 

b.  What  are  the  effects  of  the  composition  phase  plus  the  speed-up  phase  of  advanced 
learning  on  retention  of  the  skill,  relearning  of  the  skill,  and  transfer  to  related  tasks? 


c.  What  are  the  effects  of  automatization  on  retention,  relearning,  and  transfer? 

4.  Automatization  Is  a  particularly  Interesting  and  probably  Important  phase  of  advanced 
learning.  What  evidence  can  be  obtained  from  parallel  processing  with  automatized  skills?  This 
question  can  perhaps  be  answered  by  determining  whether  differences  exist  In  automatization  for 
different  types  of  tasks  (e.g.,  procedure  vs.  organized  motor)  and  possibly  for  different  numbers 
of  tasks  conducted  simultaneously. 


Measuring  Task  Difficulty  as  Mental  Effort 


The  question  of  difficulty  of  the  various  tasks  making  up  Air  Force  jobs  and  occupations  Is 
one  which  affects  the  planning  of  training  and  personnel  management.  Tasks  of  higher  difficulty 
are  assigned  to  advanced  levels  of  AFSs  and  are  usually  assumed  to  require  different  patterns  of 
aptitude  predictors  from  those  associated  with  lower  difficulty*  Differential  task  difficulty 
also  Influences  the  planning  of  training,  In  terns  of  durations  of  instructional  modules  and 
courses.  In  these  various  uses,  "task  difficulty”  is  a  variable  derived  from  estimations  by 
on-the-job  supervisors.  Added  precision  may  be  expected  from  the  development  of  direct  measures 
of  task  difficulty,  derived  from  observations  of  performance.  Such  a  method  would  make  possible 
validation  of  subjective  estimates  of  task  difficulty. 

A  number  of  Investigators  of  human  learning  and  performance  accept  the  Idea  that  human  tasks 
vary  In  the  proportion  of  limited-capacity  central  processing  they  demand  (Johnston  A  Uhl,  1976; 
Kahneman,  1973;  Moray,  1967;  Tyler,  Hertel,  McCallum,  A  Ellis,  1979).  The  central  processing  Is 
often  referred  to  as  "attention,”  and  the  allocation  of  the  resource  as  "division  of  attention” 
(Shulman,  Greenberg,  A  Martin,  1971).  An  Individual  may  need  to  divide  attention  when  required 
to  perform  a  primary  and  a  secondary  task  at  one  time.  The  degree  to  which  the  secondary  task 
performance  Is  diminished  Is  often  taken  as  a  measure  of  the  attentlonal  demands  of  the  primary 
task  (Kerr.  1973). 

As  an  example,  Tyler  et  al.  (1979)  studied  attentlonal  demands,  which  they  called  'cognitive 
effort,”  In  two  primary  tasks.  One  task  was  anagrams.  In  which,  for  example,  the  word  "doctor" 
was  transformed  Into  "dortoc”  (low  effort)  and  "croodt"  (high  effort).  A  second  primary  task 

used  sentences  to  be  completed:  "The  girl  was  awakened  by  the  frightening _ (low 

effort)";  and  "The  man  was  alarmed  by  the  frightening _  (high  effort)."  The  secondary 

task  used  In  this  study  required  the  Individual  to  respond  as  rapidly  as  possible  by  turning  off 
a  tone  delivered  through  headphones  (simple  reaction  time).  The  tone  came  on  randomly  at  four 
different  Intervals  following  the  presentation  of  each  Item  of  the  primary  task.  Findings  of  the 
study  were  that  reaction  time  was  greater  for  high  effort  than  for  low  effort  tasks. 


Measures  of  Task  Difficulty 


The  study  by  Tyler  et  al.  (1979),  as  well  as  other  studies  (Kerr,  1973;  Posner  A  Boies, 
1971),  Implies  that  the  secondary  task  oforeactlon  time  to  tone  can  be  employed  as  a  measure  of 
"mental  effort,"  and  therefore,  by  Implication,  as  a  measure  of  task  difficulty.  Presumably, 
such  a  measure  answers  the  question,  "How  much  (what  proportion  of)  attentional  resource  must  be 
devoted  to  the  performance  of  this  task?" 

It  may  be  noted  that  the  use  of  a  reaction-time  measure  of  mental  effort  Is  most  appropriate 
for  tasks  that  can  be  presented  Item  by  Item  (as  In  word  recognition,  numerical  calculation, 
concept  Identification).  Such  a  measure  would  not  be  appropriate  for  a  task  having  a  serial 
nature  In  which  attention  had  to  be  maintained  over  a  period  of  time  during  which  several 
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different  kinds  of  processing,  or  several  successive  steps  of  processing,  were  being  carried 
out.  Johnston,  Greenhag,  Fisher,  and  Martin  (1970)  employed  continuous  compensatory  tracking  of 
an  oscilloscope  signal  as  a  secondary  task;  their  primary  task  was  a  list  of  words  to  be 

categorized.  In  this  case,  tracking  error  was  used  as  the  measure  of  the  attentlonal 

"difficulty*  of  the  primary  task. 

In  the  case  of  a  procedural  task,  such  as  "adjusting  a  crystal  oscillator,"  there  are  several 
steps  which,  although  performed  one  by  one,  constitute  a  coherent  sequential  performance.  It  Is 
conceivable  that  each  step  might  be  accompanied  by  a  secondary  reaction-time  task,  yielding 
Individual  subtask  measures  that  might  be  summed  to  provide  a  total  task  measure  of  mental 

effort.  Alternatively,  If  a  simulated  version  of  the  task  was  being  presented  on  a  display 

screen,  a  tracking  task  could  be  employed  In  a  secondary  role  to  provide  a  tracking  error  score 
as  a  measure  of  mental  effort.  Independent  of  the  particular  measure  employed.  It  Is  surely 
appropriate  to  note  the  factor  of  nunber  of  steps  (cf.  Shields,  Goldberg,  &  Dressel,  1979)  as  a 
contributor  to  task  difficulty. 


Research  Questions 


1.  For  a  procedural  task  of  4  to  7  steps  to  be  carried  out  without  an  external  memory  aid 
(e.g.,  putting  an  electronic  device  into  operation),  what  Is  the  mental  effort  at  three  different 
stages  of  task  practice?  It  may  be  supposed  that  when  the  task  Is  first  done,  mental  effort  may 
be  high.  After  a  few  trials  of  practice,  the  effort  may  be  reduced  and  after  still  more 
practice,  reduced  further.  Using  the  reaction-time  technique  of  secondary  task  measurement,  a 
tone  could  be  Introduced  after  the  beginning  of  each  step  of  the  task,  perhaps  using  random 
Intervals  In  the  manner  of  Tyler  et  al.  (1979).  Such  a  method  would  relate  degree  of  training 
with  mental  effort.  In  addition,  this  method  would  yield  a  measure  of  the  difficulty  of  each 
step,  data  which  may  contribute  to  an  understanding  of  "difficulty"  as  applied  to  the  task  as  a 
whole. 


2.  For  the  same  type  of  task.  Is  difficulty  assessed  by  mental  effort  Included  In,  or 
Independent  of,  a  measure  of  task  difficulty  which  Is  "time  to  learn"?  Time  to  learn  could,  of 
course,  be  measured  by  recording  either  (a)  number  of  trials  or  (b)  elapsed  time  required  to 
reach  some  specified  criterion  of  performance. 

3.  For  the  same  type  of  task,  what  relations  exist  among  the  measures  of  difficulty  called 
(a)  mental  effort,  (b)  time  or  trials  to  learn,  and  (c)  rated  difficulty  as  given  by  the  COOAP 
system? 

4.  For  a  type  of  task  requiring  the  learning  and  differentiation  of  a  number  of  concepts, 
the  questions  posed  In  1,  2,  and  3  may  be  asked.  Examples  of  this  kind  of  task  are  the  military 
map  symbols  employed  by  Fox,  Taylor,  and  Caylor  (1969)  and  the  Power  Station  Operator  Task  used 
In  a  current  AFWL  effort.  Alternatively,  It  should  be  possible  to  identify  a  component  of  an 
Air  Force  Specialty  Code  (AFSC)  which  could  be  represented  in  a  laboratory  setting,  either 
directly  or  by  simulation.  (See  the  section  on  "Benchmark  Tasks  for  Learning  Research.") 

5.  Another  question  of  Importance  Is  the  relation  between  one  or  more  measures  of  mental 
effort  and  amount  of  retention  over  the  Interval  of  a  week  or  more.  A  summary  of  recent  research 
on  this  problem  Is  given  by  Rose,  McLaughlin,  and  Felker  (1981)  who  point  out  the  limitations  of 
these  studies. 

6.  Discovering  how  these  different  measures  of  mental  effort  relate  to  each  other  In 
learning  and  In  retention  should  make  possible  a  choice  of  a  measure  to  represent  "task 
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difficulty.*  The  application  of  the  measures  to  at  least  two  different  kinds  of  tasks  (1  and  4) 
will  make  a  beginning  In  answering  the  question  of  general izablllty.  If  a  “best  choice*  of  task 
difficulty  measures  cannot  be  made,  an  understanding  of  how  these  measures  are  related  will  be  of 
Interest. 


Skill  Maintenance  and  Skill  Forgetting 


Maintaining  the  quality  of  learned  skills  In  military  Jobs  Is  an  objective  that  forms  the 
basis  for  a  number  of  research  efforts  over  the  past  10  years  or  so  (Arlma  A  Nell,  1978; 
Goldberg,  Drillings,  A  Dressel,  1981;  Prophet,  1976;  Schendel  A  Hagnan,  1980).  Concern  with  the 
forgetting  of  skills  arises  from  a  number  of  circumstances.  Including  the  following: 

1.  Some  of  the  tasks  on  which  training  Is  given  may  not  be  used  on  the  job  Immediately  and 
tend  to  suffer  forgetting. 

2.  Skills  learned  Initially  may  not  be  required  until  5-level  (journeyman-level)  job 

proficiency  must  be  demonstrated  and  may  be  forgotten  during  that  Interval. 

3.  When  training  courses  are  of  lengthy  duration,  skills  learned  In  early  weeks  may  be 

forgotten  by  the  end  of  course. 

Forgetting  of  skills  must  be  deemed  an  unfortunate  occurrence  in  any  of  these  situations  and 
a  factor  which  Is  frequently  detrimental  to  job  performance.  The  avoidance  of  skill  forgetting 
and  the  adoption  of  procedures  to  maintain  skills  would  appear  to  be  essential  features  of  the 
total  process  of  continuing  military  readiness. 

The  forgetting  of  skills  can  presumably  be  overcome,  or  at  least  reduced  in  amount,  by 
special  precautions  taken  during  training.  Several  possible  training  variations  might  be 
considered  to  Improve  skill  retention,  according  to  previously  published  work.  Increased  amount 
of  practice,  of  course.  Is  the  best  known  suggestion  for  which  the  greatest  amount  of  evidence 
exists  (Naylor  A  Briggs,  1961;  Rose,  McLaughlin,  A  Felker,  1981).  Another  possibility  Is 
learning  “elaboration,*  as  suggested  In  the  work  of  Rohwer  (1974)  and  others.  Still  a  third  idea 

related  to  the  maintenance  of  skill  Is  a  kind  of  “tuning"  (Rumelhart  A  Ortony,  1977)  by  means  of 

learning  a  schema  or  a  “work  model"  (Bunderson,  Gibbons,  A  Olsen,  1981). 


Variables  Affecting  Skill  Retention 

Although  each  of  these  ways  of  reducing  forgetting  has  a  reasonable  basis  In  theory, 
empirical  findings  that  relate  specifically  to  theory  or  to  the  choice  of  theoretical 
alternatives  are  virtually  non-existent.  With  regard  to  the  variable  of  practice,  although  its 
relation  to  skill  retention  Is  generally  acknowledged  In  a  qualitative  sense,  quantitative 
findings  that  relate  the  amount  of  practice  to  the  amount  and  rate  of  forgetting  vary  widely. 
Although  a  number  of  studies  have  found  a  directly  proportional  relation  between  amount  of 
practice  and  amount  of  retention  (Goldberg,  et  al.,  1981;  Schendel  A  Hagman,  1980),  a  conmon 
finding  Indicates  this  relation  Is  a  decreasing  one  (Ammons  et  al.,  1958;  Jahnke,  1958). 
Mengelkoch,  Adams,  and  Gainer  (1971)  found  no  difference  In  amount  of  retention  loss  for  two 
different  amounts  of  practice  (5  and  10  trials)  for  several  different  skills  of  instrument 
flying.  In  contrast,  the  theoretical  view  of  Wlckelgren  (1981)  maintains  that  Increases  In  the 
degree  of  learning  produce  large  amplifications  In  "trace  longevity,"  even  with  no  difference  in 
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forgetting  rate.  This  conception,  however,  refers  to  what  Wlckelgren  calls  "microretrieval"  and 
so  nay  not  apply  directly  to  the  retention  of  skills. 

A  study  relating  retention  to  both  amount  of  practice  and  amount  of  attentlonal  effort 
devoted  during  practice  was  accomplished  by  Naylor,  Briggs,  and  Reed  (1968).  The  latter  variable 
was  manipulated  by  altering  the  attentlonal  demands  of  a  secondary  task,  or  what  the  authors 
named  "high  and  low  coherence."  The  primary  task  was  learning  and  retention  of  a  moving  target 
displayed  (In  three  dimensions  of  motion)  on  three  voltmeter  dials;  the  secondary  task  was 
monitoring  nine  rows  of  lights  by  pressing  buttons  which  turned  the  lights  on  and  off  according 
to  some  complex  rules.  The  variable  having  the  major  significant  effect  on  retention  was  amount 
of  training.  An  Interaction  effect  of  secondary  task  "coherence"  was  Indicated  by  the  fact  that 
retention  by  the  2-week  training  group  was  Influenced  by  this  factor,  whereas  retention  by  the 
4-week  group  was  not.  An  Interesting  finding  was  that  the  Interval  of  retention  (2  weeks  versus 
4  weeks)  did  not  produce  differential  losses  In  retention. 

A  review  of  the  Army  Research  Institute  (ARI)  research  on  retention  of  Army  skills  (Rose  et 
al.,  1981)  Identifies  a  number  of  different  task  characteristics  that  have  been  examined  for 
their  relation  to  forgetting.  Perhaps  most  Interesting  of  these  from  the  standpoint  of  future 
research  are  (a)  task  learning  difficulty,  (b)  amount  of  performance  guidance  by  cues  (KcCluskey, 
Hiller,  Bloom,  A  Whltmoush,  1978),  and  (c)  number  of  task  steps  (Shields,  Beldberg,  A  Dressel, 
1979).  These  variables  appear  to  have  effects  on  retention  In  the  expected  direction.  However, 
the  evidence  Is  weak  In  spots  and  specific  studies  of  these  factors  would  be  desirable. 

Other  findings  from  a  series  of  ARI  studies  are  these: 

1.  Significant  improvement  In  retention  results  from  practice  .beyond  the  standard 
proficiency  criterion  of  one  perfect  performance  (Goldberg,  Drillings,  &  Dressel ,  1981;  Hagman, 
1980b;  Schendel  A  Hagman,  1980). 

2.  Trials  of  practice  spaced  a  day  apart  were  found  more  effective  for  retention  than  were 
three  trials  held  one  after  the  other  (Hagman,  1980a).  However,  a  study  of  rifle  assembly  found 
no  effects  of  spacing  when  retention  was  measured  after  8  weeks  (Schendel  A  Hagman,  1980). 


Measurement  of  Skill  Retention 

On  the  whole.  It  Is  apparent  that  a  number  of  Important  research  problems  remain  If  we  are 
Interested  In  Identifying  the  variables  which  lead  to  good  retention  and  maintenance  of  skills. 
The  difficulties  encountered  In  attempting  to  Interpret  and  draw  firm  conclusions  about  skill 
retention  appear  to  have  several  basic  causes.  First,  empirical  studies  have  been  carried  out  on 
"skills"  without  agreement  on  a  common  definition  of  skill  as  a  concept.  Second,  studies  of 
forgetting  are  sometimes  Inconsistent  In  their  choice  of  measures  of  amount  of  forgetting,  rate 
of  forgetting,  and  longevity.  Related  to  this  problem  Is  the  matter  of  choice  of  ways  of 
measuring  retention.  In  particular,  measures  of  recall  and  relearning  (savings)  may  not  be  as 
simply  related  to  each  other  as  suggested  by  the  early  results  of  Luh  (1922)  on  nonsense 
syllables. 


Needed  Research 

Drawing  sensible  conclusions  from  studies  of  skill  forgetting  Is  virtually  Impossible  at  the 
present  time.  The  Interpretation  of  a  fundamental  state  of  knowledge  would  appear  to  depend  on 
the  following  kinds  of  Information,  but  which  Is  not  available: 


1.  For  a  procedure  containing  four  non-interfering  steps,  measures  of  accuracy  of  recall 
following  100S  learning  after  three  time  Intervals  (innedlate,  2  days,  2  weeks).  Each  step  In 
the  procedure  should  require  Identification  of  a  'thing"  category  and  an  “action*  catego»y  at 
least  (e.g.,  turn  knob  A  to  27").  Motor  aspects  of  the  procedure  should  be  previously  well 
learned. 

2.  Same  as  1,  after  the  time  Intervals  Innedlate,  2  weeks,  and  2  months. 

3.  Same  as  1  and  2,  for  procedures  of  4,  5,  16,  and  32  steps. 

4.  Same  as  3,  with  steps  that  are  likely  to  exhibit  Interference. 

These  observations  would  simply  be  expected  to  yield  some  baseline  data.  The  real  oroblems 
could  then  be  tackled.  In  view  of  this  purpose,  care  should  be  taken  to  choose  the  procedural 
tasks  carefully.  Their  component  steps  should  be  "realistic"  ones  (like  setting  knobs);  but  at 
the  same  time,  peculiarities  of  actual  job  components  should  be  avoided  in  the  Interest  of 
general Izable  results.  Depending  on  the  outcome  of  these  exploratory  studies,  the  following 
research  questions  can  be  undertaken: 

1.  What  happens  to  performance  of  a  task  when  practice  with  feedback  Is  continued  beyond  the 
criterion  of  one  adequate  performance?  What  effect  do  differing  degrees  of  overlearning  have  on 
retention? 

2.  What  differential  effects  on  retention  (recall)  result  from  task  “coherence"  factors,  or 
task  characteristics,  that  distinguish  procedures  from  motor  skills? 


Skill  Retention  and  Relearning 


A  number  of  reviews  have  been  made  of  studies  of  retention  of  skills  In  military  occupations 
(Hagman,  Rose,  McLaughlin,  &  Felker,  1981;  Naylor  4  Briggs,  1961;  Schendel,  Shields,  *  Katz, 
1978;  Shields,  Goldberg,  A  Dressel,  1979).  It  Is  evident  that  these  studies  have  been  successful 
In  Identifying  a  number  of  important  variables  affecting  skill  retention  and  maintenance. 
Including  level  of  performance  attained  In  learning,  spacing  versus  massing  of  practice,  and 
factors  In  task  composition. 

While  the  amount  of  retention  of  skills.  Implying  as  It  does  different  levels  of  job 
performance.  Is  obviously  Information  of  value  to  managers  of  military  operations,  a  related 
aspect  of  skill  retention  Is  at  least  of  equal  Interest.  This  Is  the  matter  of  how  quickly 
skills  can  be  relearned  following  a  period  of  disuse.  Skills  acquired  early  In  a  technical 
school  course  may  need  to  be  relearned  during  a  late  period  of  the  same  course.  Likewise,  skills 
learned  In  a  specialty  school  may  not  be  required  to  be  used  Immediately  In  a  job  assignment,  and 
so  may  need  relearning.  Supervisors  responsible  for  on-the-job  training  (OJT)  need  to  be  aware 
of  the  possibilities  of  skill  deterioration  and  of  requirements  for  retraining  of  airman  job 
1  ncumbents. 

Most  studies  of  skill  retention  assess  amount  retained  as  a  recall  score.  A  few  (Ammons  et 
al.,  1958;  Grlmsley,  1969;  Mengelkoch,  Adams,  4  Gainer,  1971;  Prophet,  1976)  have  additionally 
measured  time  on  trials  needed  to  retrain.  Traditionally,  relearning  Is  expressed  as  a  "saving 
score"  (Woodworth,  1938),  made  Into  a  percentage  by  dividing  the  time  saved  by  the  original 
learning  time  (or  trials  saved  by  the  original  learning  trials).  In  some  early  studies  of  verbal 
learning,  the  percent  of  saving  was  found  to  be  roughly  comparable  to  the  amount  recalled 


expressed  as  a  percent  (Krueger,  1929).  However,  the  saving  score  of  relearning  apparently 
differs  greatly  from  task  to  task.  For  example,  Ammons  et  al.  (1958),  Investigating  relearning 
of  a  continuous-control  motor  task  practiced  Initially  for  8  hours  following  a  period  of  6  months 
of  no  practice,  found  that  Initial  proficiency  was  regained  In  13  to  14  minutes.  In  contrast,  a 
15-step  procedural  task  practiced  Initially  for  30  trials  appeared  (by  extrapolation)  to  require 
learners  at  least  15  trials  to  regain  their  original  proficiency  level  after  the  same  length  of 
disuse.  Expressed  as  percent  savings,  retention  of  the  motor  skill  is  97%,  whereas  retention  of 
the  procedural  skill  Is  501. 

Although  relearning  has  both  theoretical  and  practical  Importance,  the  results  of  existing 
Investigations  do  not  clearly  Identify  the  variables  which  determine  Its  amount.  For  any  given 
skilled  performance,  we  are  not  able  to  predict  whether  it  will  be  relearned  rapidly  or  slowly, 
as  compared  with  its  original  learning.  From  the  existing  evidence.  It  would  appear  that  the 
following  variables  have  prominent  effects  on  relearning: 

1.  Level  of  original  learning  (or,  amount  of  original  practice). 

2.  Length  of  Interval  of  disuse. 

3.  Composition  of  the  skill. 

In  addition,  the  variable  of  amount  recalled  may  be  Identified.  What  this  means,  In  effect. 
Is  that  the  relation  between  amount  recalled  and  amount  of  relearning  required  is  not  known. 
Presumably,  this  relationship  may  differ  when  paired  with  any  of  the  variables  in  the  previous 


Research  Problems 

Besides  the  needed  additional  research  on  skill  retention  as  measured  by  recall  scores,  there 
Is  a  gap  In  knowledge  to  be  filled  by  the  investigation  of  relearning  of  skills.  The  following 
list  represents  an  approximate  sequence  for  study  of  these  problems.  Initially,  It  appears 
desirable  to  choose  an  unvarying  time  for  a  no-practice  Interval,  In  order  to  assess  the  effects 
of  other  variables.  In  later  research,  the  no-practice  Interval  may  be  deliberately  varied  in 
order  to  study  Its  effects. 

1.  Relation  of  task  composition  to  relearning.  The  composition  of  a  procedural  task  can  be 
varied  In  at  least  two  ways:  by  variation  In  the  number  of  steps  Involved  and  In  the  coherence 
of  the  task.  Coherence  is  used  in  the  study  by  Naylor,  Briggs,  and  Reed  (1968),  where  the 
definition  Is  "cueing  facilitation  from  one  step  to  the  next."  In  other  words,  a  highly  coherent 
task  Is  one  in  which  the  cues  for  each  next  step  are  obvious. 

a.  Beginning  from  a  level  of  learning,  such  as  two  performances  In  succession  without 
error,  compare  the  relearning  of  procedural  tasks  of  three  different  lengths:  12  steps,  24 
steps,  and  36  steps.  The  no-practice  interval  could  be  5  days. 

b.  With  the  same  t  eatment,  compare  the  relearning  of  procedural  tasks  of  three 
different  levels  of  coherence. 

2.  Relation  of  task  length  to  relearning.  Procedural  tasks  contain  different  numbers  of 
steps.  Again  beginning  with  learning  to  a  convenient  criterion,  compare  the  relearning  of  tasks 
of  12,  24,  and  36  steps.  The  no-practice  interval  could  be  5  days. 


.  Relation  of  level  of  learning  to  relearning.  Having  established  some  baseline  measures 
under  the  studies  of  1  and  2,  It  will  now  be  possible  to  select  one  or  more  learning  tasks  that 
can  be  used  for  studies  relating  level  of  learning  to  trials  required  for  relearning.  Although 
the  relationship  of  end-of-practlce  performance  level  to  amount  retained  (recalled)  has  been 
tested  many  times,  It  Is  not  apparent  that  the  same  relationship  will  be  found  for  relearning. 

4.  Relation  of  recall  to  relearning.  As  Implied  In  3,  the  relation  of  recall  scores  to 
relearning  measures  should  be  studied.  It  may  be  possible  to  accomplish  some  preliminary 
Investigations  of  this  relationship  In  connection  with  other  studies  of  this  sort  described  in  1, 
2,  and  3.  Once  the  constraining  factors  are  well  understood,  a  direct  Investigation  of  this 
problem  should  be  possible.  The  central  question  Is  as  follows:  Under  what  circumstance,  and 
for  what  kinds  of  tasks.  Is  amount  of  required  relearning  predictable  from  measures  of  recall? 

5.  Relation  of  relearning  to  length  of  no-practice  Interval.  While  virtually  all  of  the 
studies  described  previously  can  be  conducted  with  some  disuse  Interval  of  a  single  length  (such 
as  5  days),  attention  turns  here  to  the  “forgetting  Interval"  variable  Itself.  The  various 
factors  outlined  In  previous  sections  (task  variables,  learning  variables)  will  probably  be  found 
to  Interact  with  the  forgetting-lnterval  variable,  so  that  no  simple  answer  to  this  question  Is 
likely. 
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CHAPTER  4 


TRAINING  EXPERT  APPRENTICES 


Robert  Glaser 

Learning  Research  and  Development  Center 
University  of  Pittsburgh 


What  are  some  fundamental  considerations  In  the  design  of  an  Air  Force  laboratory  for 
research  In  learning  and  training? 

I  believe  that  the  laboratory  should  focus  on  the  question  of  how  to  produce  expert 
apprentices.  By  "expert  apprentices"  I  mean  people  who,  when  they  leave  training,  have  strong 
capabilities  to  profit  fro*  the  experience  they  receive  In  their  Military  careers.  What  kind  of 
basic  and  more  advanced  knowledge  and  skills,  what  kind  of  learning  skills,  what  perception  of 
their  jobs  should  they  be  given  In  school  so  that  they  can  profit  maximally  from  later  experience? 

There  has  always  been  a  debate  about  the  relationship  between  what  students  learn  In  school 
and  what  they  need  for  the  job;  school  Is  just  school,  and  experience  Is  necessary.  So  It  seems 
to  me  the  mission  of  school  should  be  to  produce  people  who  are  highly  capable  of  profiting  from 
work  In  the  field.  That  Is  what  I  mean  by  producing  expert  apprentices;  l.e.,  novices  who  have 
an  adequate  Information  base  and  skills  for  learning  new  things.  Apprentices  can  do  many  things, 
but  they  are  not  experts;  they  are  far  from  experts,  and  on-the-job  training  and  experience  can 
take  place  efficiently  with  appropriately  trained  apprentices. 

A  key  objective  of  schooling  Is  to  train  people  In  abilities  that  enable  them  to  profit  from 
the  experience  after  school.  The  differences  In  the  knowledge  structures  and  perceptions  of 
novices  and  people  who  have  some  degree  of  experience  are  so  drastically  different  that  we  have 
to  put  our  apprentices  in  a  position  where  they  can  acquire  most  readily  the  kind  of  teaching 
that  comes  fro*  experience.  Apprentices  need  to  learn  as  they  work  In  their  field  units.  We 
want  to  maximize  this  benefit  through  what  Is  taught  In  Initial  training. 

In  order  to  think  through  this  problem,  we  first  have  to  ask  what  It  is  that  is  acquired  by 
experience,  and  then  we  have  to  see  whether  It  Is  possible  to  do  things  In  training  that  maximize 
what  students  are  supposed  to  get  out  of  experience.  That  Is  the  line  of  this  logic.  First,  I 
discuss  what  1$  acquired  by  experience  and  then  list  suggestions  for  research. 


What  Is  Acquired  Through  Experience? 


Categorization  Ability 


People  who  are  experienced  are  very  good  categorizers;  that  is,  when  meeting  up  with  a 
situation,  they  encode  It  into  a  category  of  knowledge  that  has  included  In  it  ways  of  handling 
or  solving  the  problem.  Experts  are  excellent  categorizers,  as  shown  In  the  expert-novice  chess 
studies.  These  experts  do  not  have  super  processing  powers.  They  recognize  a  configuration  of 
pieces,  and  that  configuration  has  meaning  -  like  a  word  has  meaning  -  and  they  know  what  to  do 
with  it.  Through  experience,  they  become  expert  at  recognizing  and  categorizing  things;  and 
there  is  a  great  deal  of  automatlclty  to  this  performance.  For  example,  an  experienced  jet 
engine  mechanic  rapidly  categorizes  a  reported  engine  malfunction  into  a  class  of  repairs  that 
has  certain  characteristics  and  not  others  (e.g.,  core  engine  damage  requiring  1  week  of  In-shop 
repair);  an  experienced  manager  classifies  a  loss  of  morale  as  requiring  departmental 


reorganization  rather  than  personnel  dismissal.  In  both  of  these  cases,  features  of  the  problem 
situation  result  In  the  accessing  of  categories  In  memory  that  are  attached  to  probable  solution 
procedures. 


Categorization  has  certain  additional  properties.  It  Involves  sorting  on  the  basis  of 
"principled"  knowledge  versus  what  can  be  called  "literal"  knowledge.  In  the  physics  studies, 
for  example,  when  a  novice  Is  asked  to  look  at  a  problem,  he  or  she  says  It  Is  a  pulley  problem, 
or  It  Is  an  Inclined  plane  problem;  but  when  you  ask  an  experienced  Individual  to  look  at  the 
problem,  he  or  she  says  It  Is  a  Newton's  Second  Law  problem,  or  it  Is  a  force  problem.  There  Is 
a  relatively  automatic  categorization  of  the  problem,  but  It  Is  based  on  principled  knowledge 
rather  than  literal  feature  knowledge.  This  Is  the  kind  of  proficiency  that  should  be  encouraged 
by  experience.  The  question  Is:  What  basis  for  It  might  be  supplied  In  training?  This  is  the 
problem  I  am  trying  to  pose.  Principled  knowledge  Is  acquired  both  In  school  training  and  as  a 
result  of  on-the-job  experience.  However,  school  training  can  provide  a  basis  that  fosters  or 
inhibits  Its  development;  e.g.,  the  kind  of  understanding  of  electronic  principles  that  an 
Individual  acquires  In  training  will  Influence  the  way  In  which  experience  with  on-the-job 
troubleshooting  problems  Is  organized  for  further  use. 


Mental  Models 

Another  thing  that  people  gain  from  experience  Is  adequate  theories  or  mental  models  of  their 
work;  they  have  especially  effective  ways  of  representing  the  situations  that  confront  them. 
Highly  proficient  people  have  models  that  work,  schemata  that  are  appropriate;  and  these  working 
theories  are  more  complete  than  those  of  the  less  experienced.  These  theories  or  models  that 
people  have  In  their  minds  are  used  as  scientists  use  theories  when  they  think  about  a  problem. 
They  use  a  theory  to  make  predictions,  to  think  with,  and  to  say  whether  something  else  should  be 
present  in  the  situation  that  Is  not  apparent.  A  theory  of  a  task  works  like  a  cognitive  schema; 
it  is  a  structure  of  knowledge  that  Is  used  to  think  with  and  to  remember  things.  Schemata  or 
models  (or  organizations  of  knowledge)  are  constantly  amended  through  on-the-job  experience. 
Models  of  knowledge  are  challenged  In  the  course  of  experience.  If  the  model  does  not  conform  to 
real  situations,  changes  are  made  or  reality  Is  assimilated  to  the  model.  When  novices  come  into 
a  situation,  they  too  have  mental  models  -  naive  ones  which  are  relatively  Incomplete  and 
Inadequate  -  but  it  Is  these  theories  or  knowledge  organizations  that  have  to  be  addressed  and 
developed  through  appropriate  teaching. 


Automatization 

Experts  display  a  good  deal  of  very  rapid  performance  without  apparent  cognitive  effort. 
They  "parallel  process"  such  that  various  attentional  requirements  do  not  Interfere  with  one 
another;  highly  skilled  components  of  performance  occur  with  little  apparent  conscious 
processing.  In  particular,  the  basic  skills  of  the  expert  are  automatized,  such  that  mental 
space  is  left  over  for  aspects  of  performance  that  require  conscious  processing.  (Like  tennis 
professional s,  they  have  their  basic  strokes  well  practiced  so  they  do  not  have  to  worry  about 
their  backhand  and  can  concentrate  on  the  strategy  of  the  game.)  Another  dimension  of  training 
is  to  assist  in  the  development  of  automaticity  in  basic  skills  so  that  the  time  left  over  for 
higher-level  job  performance  is  maximized. 
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Procedure lization 


The  knowledge  of  an  expert  Is  procedural  1  zed.  Experts  not  only  know  things,  but  they  know 
when  that  knowledge  Is  applicable;  they  know  when  it  is  to  be  used  and  when  It  is  not  to  be 
used.  The  expert's  knowledge  is  embedded  in  procedures  for  the  use  of  that  knowledge.  Expert 
solvers  of  physics  problems  not  only  can  identify  a  Newton's  Second  Law  problem,  but  they  can 
also  can  Indicate  the  conditions  under  which  a  form  of  Newton's  Second  Law  Is  applicable.  Their 
knowledge  is  procedural  1  zed  by  being  attached  to  a  set  of  conditions  for  its  use.  (If  you  think 
in  terms  of  production  system  statements  such  as  "If-then,"  then  you  know  that  the  expert's 
knowledge  not  only  has  the  "then,"  but  also  has  the  “if"  attached  to  it.  The  novice's 
performance  Is  less  active.  It  Is  largely  "then."  There  Is  not  much  "If"  related  to  it.)  If 
the  conditions  are  such  and  such,  then  you  do  that;  this  is  what  I  mean  by  procedural  1  zed 
knowledge.  The  training  question  then  becomes:  How  do  we  produce  knowledge  in  novices  so  that 
it  becomes  procedural  1 zed? 


Metacognition  and  Learning  Skill 


Another  thing  that  Is  Important  to  acquire  in  training  is  the  capability  to  learn  from 
experience.  We  have  to  understand  what  this  capability  is.  To  some  extent,  it  consists  of  a 
capability  for  monitoring  one's  own  learning,  or  of  Interrogating  learning,  so  that  the 
experience  obtained  can  be  managed  and  assimilated.  It  is  this  kind  of  metacognitive  skill  that 
good  learners  display.  When  they  hear  something,  they  check  on  it,  they  try  to  reconcile  it  with 
the  job  schemata  they  have,  and  they  check  on  the  products  of  their  problem  solving.  Mature 
learners  acquire  these  capabilities,  and  an  objective  of  training  Is  to  help  novices  begin  to 
develop  the  same  capabilities. 


Training 


The  above  performance  factors  are  what  experts  learn  after  time  on  the  job,  and  the  objective 
of  training  is  to  produce  people  who  profit  from  job  experience  so  that  they  develop  these 
characteristics  of  expertise.  Now  comes  the  crucial  task:  What  do  you  do  in  the  formal  training 
in  the  military  to  achieve  this? 

There  are  cycles  of  training,  first  on-the-job  and  then  back  to  retraining,  but  a  stage  can 
be  defined  as  training  and  then  on-the-job  training.  Therefore,  the  research  issue  becomes  how 
to  design  training  and  research  on  training  (within  a  stage)  that  addresses  these  objectives  of 
developing  expertise.  How  can  training  encourage  the  capability  to  profit  from  experience  that 
yields  high  levels  of  proficiency?  One  clue  has  already  been  given  to  us  by  what  Ray  Christal 
said.  He  is  thinking  about  assessing  automatization  or  the  progress  in  automatization.  In  this 
regard,  the  work  by  Wally  Schneider  (1982)  on  automatization  is  very  interesting.  He  indicated 
that  the  way  to  produce  "unconscious  processing"  is  to  find  some  consistency  of  performance, 
because  inconsistency  requires  conscious  processing.  Schneider  studies  how  air  traffic 
controllers  determine  Intercept  paths.  The  tasks  look  varied,  but  he  found  a  consistent 
principle  that  these  people  can  use  and  that  can  be  repetitively  practiced  under  speed  stress,  so 
that  it  becomes  highly  automatized. 

The  idea  is  to  determine  the  conditions  that  produce  automaticity  and  ways  of  assessing  it  to 
Insure  its  reliable  presence.  It  is  possible  that  when  basic  skills  are  measured  when  people 
leave  school,  a  proficiency  examination  should  assess  the  degree  to  which  some  of  their  knowledge 
or  skills  have  been  automatized.  For  example,  long-term  studies  of  reading  show  that  students 
who  develop  automatized  skills  in  the  early  grades  may  be  those  who  in  the  long  run  develop 
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comprehension  skills  faster.  This  suggests  that  during  training,  assessments  could  be  made  of 
the  level  of  automatization  characteristic  of  those  students  Mho  develop  better  and  more  complex 
knowledge  later  In  training  or  on  the  job. 

Problem  Solving  and  Knowledge 

Learning  skills  and  problem-solving  skills  can  be  developed  and  strengthened  In  the  course  of 
training.  In  the  course  of  acquiring  Information  and  skill,  students  can  Interrogate  a  body  of 
knowledge,  solve  problems  about  the  application  of  their  knowledge,  make  up  theories  and 
extrapolate  from  the  Information  they  have.  Students  have  models  of  what  they  are  doing,  and  the 
opportunity  to  Interact  with  the  model  develops  the  capability  for  interrogating  their  own 
knowledge  In  a  way  that  leads  to  learning. 

There  currently  are  a  number  of  widely  used  training  programs  based  on  information-processing 
psychology  and  older  psychometric  theory  that  attempt  to  train  people  in  general  problem-solving 
heuristics.  However,  the  theory  on  which  these  programs  Is  based  Is  changing.  New  theory 
emphasizes  the  Importance  of  learning  problem-solving  as  a  person  acquires  knowledge.  For 
example.  In  the  artificial  Intelligence  field,  there  has  been  a  shift  from  building  machines  with 
big  search  algorithms  that  could  search  faster,  with  quicker  hardware,  to  the  building  of  better 
knowledge  structures  that  can  be  readily  accessed.  Knowledge  and  process  are  highly  Interactive 
in  human  performance. 

Knowledge  structure  interacts  with  memory  skills.  For  example,  in  a  study  of  chess 
knowledge,  the  Investigator  took  children  who  knew  chess  well  and  adults  who  did  not,  and  gave 
them  two  memory  tasks:  (a)  remembering  a  briefly  presented  chess  board  and  (b)  performing  a 
digit  span  task  of  the  kind  that  appears  on  standardized  intelligence  tests.  The  results  showed 
that  the  children  had  very  good  memories,  and  the  adults  had  poor  memories,  for  chess  positions. 
On  the  digit  span  test,  however,  the  children  had  much  more  limited  memory  than  did  the  adults 
(the  usual  test  finding  on  the  knowledge-lean  digit  task).  Therefore,  there  are  interactive 
effects  of  knowledge  and  process  that  have  to  be  considered  In  training.  Knowledge  Is  not  just 
Inert,  waiting  to  be  used;  rather,  knowledge  Is  an  active  shaper  of  cognitive  process. 

These  are  very  complex  problems,  because  It  Is  obvious  that  some  people  are  very  good  at 
employing  generalized  processes  when  they  confront  a  situation  where  they  do  not  have  any  or  much 
knowledge.  However,  when  they  confront  a  situation  where  they  have  knowledge  that  can  be  brought 
to  bear,  they  do  not  use  these  general  heuristics;  rather,  they  employ  knowledge-specific 
processes  that  are  more  powerful  in  particular  situations  than  the  more  general  methods.  Thus, 
problem  solving  and  proficiency  In  higher-level  processes  are  best  acquired  in  the  context  of  a 
knowledge  base.  The  kind  of  instruction  that  this  implies  is  not  unlike  the  kind  being  thought 
about  in  Intelligent  computer  systems,  w  ere  people  can  constantly  interrogate  and  inquire  in 
order  to  learn. 

There  is  a  good  article  by  Collins  and  Stevens  (1982)  that  analyzes  the  method  of  inquiry 
teachers.  What  do  inquiry  teachers  do?  They  force  the  student  to  generalize;  they  give 
counterexamples;  they  reduce  an  argument  to  a  Reductlo  Ad  Absurdum  to  show  it  is  not  a  complete 
case.  Inquiry  instructors  use  entrapment  strategies.  They  try  to  make  explicit  the  incomplete 
knowledge  of  students  by  forcing  them  to  apply  It  in  carefully  selective  situations.  This  form 
of  interactive  instruction  develops  students'  attitudes  toward  interrogating  their  knowledge. 
They  learn  to  reason  and  to  come  up  with  theories  about  what  they  think  they  know.  They  ask  the 
instructor  to  interrogate  them  or  interrogate  their  knowledge  base  themselves  so  that  their  ideas 
get  changed.  In  this  way,  students  develop  a  capability  for  flexibility  in  which  their  knowledge 
can  be  amended  in  one  way  or  another  in  the  course  of  training  and  job  experience. 


Halve  and  Pedagogical  Theories 


At  the  beginning  of  Instruction,  students  have  some  kind  of  a  beginning  knowledge  structure 
and  a  theory  of  this  knowledge  that  has  to  be  taken  Into  consideration.  One  way  to  do  this  Is  to 
design  "pedagogical11  theories  that  facilitate  learning.  The  Instructor  gives  the  student  a 
theory  that  Is  easy  to  manage  -  a  theory  that  can  be  used  to  remember  and  access  knowledge,  to 
exercise  knowledge,  to  work  with  knowledge  -  but  which  Is  not  necessarily  an  accurate  theory. 
(It  Is  a  pedagogical  theory  that  will  soon  be  overthown. )  Instructors  can  present  such  theories 
to  a  class  or  to  an  Individual.  Good  teachers  do  this  all  the  time.  They  say,  for  example,  "Let 
us  assume  that  what  we  are  learning  operates  this  way."  This  tactic  gives  students  a  structure 
to  think  with,  but  the  teacher  knows  full  well  that  It  Is  only  an  Interim  pedagogical  theory.  It 
seems  to  me  that  teachers  (rather  than  saying,  as  many  teachers  do,  when  a  student  has  trouble  in 
learning,  "Let  me  give  you  an  example")  should  say  ‘Let  me  give  you  a  theory."  They  should  say, 
"Let  me  give  you  a  structure  In  which  you  can  put  what  you  know."  I  do  not  mean  to  be  too 
sophisticated  when  I  use  the  word  "theory."  For  example,  people  have  a  model  or  theory  of  the 
task  for  a  series  of  procedures  when  they  operate  a  piece  of  equipment. 

At  the  beginning  of  Instruction,  people  have  theories  that  are  very  persistent.  There  are 
studies  on  "naive  physics"  theories  that  students  believe.  These  Investigations  took  college  and 
high  school  students  and  described  situations  to  them  (e.g.,  a  steel  ball  In  a  tube  going  around 
In  a  spiral)  and  asked  them  questions  (e.g.,  what  direction  the  ball  would  continue  to  go  when  it 
emerged  from  the  tube).  A  variety  of  situations  like  this  were  presenter  The  answers  the 
students  gave  were  Inadequate  Aristotelian  theories  of  motion  (which  are  Intuitively  what  people 
think  should  happen),  but  were  not  what  really  happens  In  Newtonian  mechanics.  The  actual 
theories  of  physics  are  very  nonlntultl ve,  and  students  come  to  Instruction  with  naive  theories 
that  a  teacher  needs  to  understand  In  order  for  Instruction  to  proceed  properly.  Studies  have 
been  carried  out  on  the  kind  of  theories  of  electronic  circuits  with  which  students  come  to 
electronics  training. 

In  physics  It  Is  possible  for  students  to  come  from  a  course  with  a  whole  new  set  of  words 
and  some  ability  to  solve  physics  equations,  but  still  believing  In  the  old  theory.  The 
significant  question  for  training  research  Is  how  to  produce  novices  who  have  reasonably  useful 
theories  that  they  can  use  to  learn  from  experience.  Theories  can  be  taught  as  entitles  of 
explanation,  along  with  a  description  of  the  operation  of  a  system.  Theories  can  also  be  taught 
as  models  that  can  be  questioned  and  tested  against  certain  obtained  Information.  This  latter 
form  of  Instruction  encourages  new  experiences  to  be  assimilated  into  integrated  knowledge 
structure.  Theory  developed  In  this  way  results  In  better  memory  storage  and  accessibility  of 
practical  procedures  associated  with  particular  theories  of  operation.  (See  the  earlier  Problem 
Solving  and  Knowledge  section  of  this  paper  and  the  research  suggestions  section  at  the  end.) 


Theory  end  Practice  In  Training 


One  has  to  look  very  carefully  at  the  various  occupational  specialties  In  the  Air  Force  and 
begin  to  ask  whether  theory-based  Instruction  Is  useful  for  best  understanding  and  performance  of 
a  job.  We  need  to  concentrate  on  the  development  of  rapid  automatization  and  less  conscious 
processing  In  certain  areas  of  a  job.  For  others,  there  are  the  kinds  of  decisions  that  take 
thinking  time,  so  there  should  be  a  balance  in  certain  job  situations  between  teaching  in 
relation  to  theories  and  models  and  teaching  In  a  form  of  drill  and  practice.  This  is 
complicated  because  even  simple  operating  procedures  are  often  best  performed  if  one  has  a  theory 
or  model  In  mind.  There  are  studies  that  show  that  procedures  are  better  carried  out  when  one 
begins  with  some  kind  of  advance  organizer  rather  than  Immediately  with  the  first  step.  In  an 


assembly  task,  rather  than  doing  step  one,  step  two,  etc..  It  say  be  better  to  start  with  the 
Idea  of  where  the  assembly  Is  to  end  up,  so  that  some  kind  of  theory  guides  performance. 

Thus,  different  job  specialties  have  various  dimensions,  and  training  questions  arise.  For 
example.  Is  It  worthwhile  to  use  Interactive  model-driven  forms  of  Instruction,  or  Is  an 

Interactive  form  of  Instruction  too  time-consuming  for  certain  jobs?  This  Is  a  matter  for 
training  research. 

It  Is  useful  for  training  research  to  consider  the  conflict  between  theory  and 
automatization.  For  example,  In  learning  mathematics  there  are  conflicting  demands  for 
mathematical  understanding  and  computational  skill.  Recent  research  Is  making  clear  that  to 
compute  with  numbers  (In  word  problems  for  example)  one  has  to  have  certain  forms  of 

understanding  and  that  the  debate  between  computational  skill  and  understanding  Is  not  a  useful 
discussion  to  continue.  In  mathematics  and  other  subject  matters,  understanding  and  skill  are 

mutually  supportive  and  are  not  an  all-or-none  matter.  Understanding,  such  as  knowledge  of  the 

function  of  place  value  In  numbers,  makes  one  better  In  subtraction,  encourages  faster  learning 
and  results  In  fewer  bugs  In  the  course  of  learning. 

Understanding  and  learning  are  related  to  "naive  theory"  which  has  been  mentioned  above. 
"Naive  theory"  can  be  defined  as  the  Intelligent  systematlclty  of  the  beginning  learner.  This  Is 
the  basis  for  "buggy  analysis"  by  some  researchers.  Based  on  this  work,  an  Important  kind  of 
research  for  students  coming  Into  the  Air  Force  In  a  certain  training  program  Is  to  make  a 
collection  of  the  systematlcltles  of  thinking  that  these  students  bring  to  training.  What  kinds 
of  bugs  do  they  generate  In  the  course  of  learning?  A  catalog  should  be  made  of  these  Initial 
conceptions  so  that  these  Ideas  can  be  treated  as  forms  of  thinking  and  not  as  stupid,  lazy 
errors.  Students  are  Intelligent  people  and  have  forms  of  thinking  that  need  completion.  One 
example  Is  In  writing,  where  students  have  a  conflict  between  their  naive  spoken  language  and  the 
formal  demands  of  written  language.  In  the  course  of  this  they  develop  a  kind  of  metalanguage 
that  Is  between  the  two.  This  results  In  forms  of  writing  prose  that  are  forms  of  one's  speech 
and  are  Intelligent  ways  of  handling  oral  language,  but  which  are  not  Intelligent  ways  to  handle 
the  formal  writing  system.  If  these  are  understood  as  systematic  errors,  the  Instructor  can  work 
with  systematic  ways  of  Interrogating  the  students'  conceptions.  The  Instructor's  attitude  can 
be  that  the  student  has  a  form  of  knowledge  or  a  theory  that  Is  Incorrect  and  that  can  be 
repaired  through  appropriate  training. 

In  general,  problems  In  training  should  be  driven  by  the  objective  of  developing  intelligent 
apprentices.  The  objectives  Involved  are  those  described  above.  These  dimensions  of  expertise 
should  influence  Instruction  and  research  on  training. 


Macroprocess  In  Training  (School  and  Classroom  Variables) 


So  far  we  have  been  considering  what  can  be  called  the  microprocesses  of  learning  or 

cognition  as  these  relate  to  proficiency  In  performance.  There  is  also  a  kind  of  work  in 

training  research  that  involves  the  macroprocesses  of  instruction.  The  macroprocesses  of 

Instruction  are  the  teaching  and  management  variables  of  the  school  and  classroom.  These  Include 
variables  such  as:  time  on  task.  Interaction  between  student  and  teacher,  and  form  of 

examination.  While  the  macroprocess  of  teaching  and  the  microprocess  of  learning  Interact,  much 
progress  can  be  made  by  operating  on  the  macro  level.  Just  as  the  steam  engine  was  built  without 
a  theory  of  thermodynamics,  we  need  to  become  more  effective  In  work  at  the  macro  level.  There 
Is  a  long  history  of  studies  of  good  and  bad  teaching  that  are  not  very  Informative,  but  this 
field  is  becoming  more  sophisticated  and  Is  beginning  to  look  at  classroom  processes  in  more 
potentially  useful  ways.  For  example,  a  recent  research  project  has  analyzed  video-tape  records 


25 


of  good  and  poor  teachers,  In  terns  of  the  planning  structure  methodology  of  artificial 
Intelligence.  In  these  tapes,  you  see  teachers  conducting  a  lesson  In  mathematics  and  can 
examine  the  details  of  student  Interaction  and  lesson  management.  The  planning  structure  of  the 
good  teacher  Is  analyzed  In  detail  and  compared  with  the  planning  structure  of  the  less 
experienced  teacher.  Certain  activities  become  apparent,  such  as  subtle  forms  of  Individualizing 
and  adapting  to  student  requirements  In  the  context  of  group  Instruction.  These  data  would  be 
very  useful  for  Instructor  training. 

These  macrovarlables  of  teaching  such  as  classroom  management,  the  design  of  the  motivational 
environment,  peer  help  In  the  classroom  and  the  other  such  factors.  If  attended  to  In  research, 
might  well  result  In  a  better  understanding  of  the  dynamics  of  the  task  of  the  Instructor  In  the 
classroom.  Thus,  I  would  advocate  research  on  both  the  macrovarlables  just  described  and 
cognitive  mlcrovarlables  previously  mentioned. 


Research  and  Devel< 


How  should  scientists  relate  to  development?  Please  remember  that  knowledge  In  the  field  of 
behavioral  science,  for  the  most  part.  Is  not  placed  In  rigorous  enough  form  that  It  can  be 
transmitted  In  a  way  which  Is  not  subject  to  possible  misinterpretation  In  development.  There 
are  subtleties  and  ambiguities  Involved.  A  research  psychologist  can  say  that  I  know  something 
about  the  acquisition  of  vocabulary  and  text  comprehension,  and  that  as  a  result  of  this 
Information  It  seems  to  be  a  good  Idea  to  design  Instruction  In  this  way.  When  someone  tries  to 
pick  this  up,  they  frequently  caricature  the  researcher's  Idea.  There  was  some  documentation  on 
this  matter  In  the  early  years  of  the  space  program  when  It  was  found  that  If  the  scientist  who 
developed  an  original  Idea  participated  In  some  way  through  the  stages  of  development,  then  these 
stages  were  carried  through  with  more  efficiency  and  success.  At  least  the  opportunity  for  the 
researcher  to  observe  the  Implementations  of  the  original  Ideas  appears  to  be  an  important  aspect 
of  the  research  and  development  continuum. 


Research  Suggestions 


In  this  section,  I  list  research  Ideas  related  to  sections  of  this  paper  concerned  with  what 
Is  acquired  through  training  and  the  conduct  of  the  training  process.  Research  questions  or 
topics  are  briefly  listed,  and  reports  of  relevant  research  are  given  that  illustrate  how 
questions  might  be  formulated  and  Indicate  methodologies  of  approach  that  might  be  used.  The 
reports  referred  to  are  In  the  reference  list  that  follows. 


Categorization  ability.  How  Is  knowledge  categorized  and  how  do  these  categories  change  as  a 
result  of  training  and  experience?  What  Is  the  difference  between  novices  and  experts,  and  how 
might  knowledge  of  this  difference  Indicate  how  training  should  proceed?  Representative  articles 
Include  Chi,  Feltovlch,  and  Glaser  (1981);  Chi,  Glaser,  and  Rees  (1982);  Egan  and  Schwartz 
(1979);  and  Paige  and  Simon  (1966). 


Principled  knowledge.  In  addition  to  the  procedures  required  for  performing  a  task,  what 
perceptions  and  understanding  of  the  task  must  be  learned  to  ensure  that  these  procedures  can  be 
carried  out  rapidly  with  a  minimum  of  error?  A  representative  article  by  Riley,  Greeno,  and 
Heller  (1983)  examines  children's  problem-solving  ability,  but  the  methodology  could  be  very 
useful  to  consider  for  technical  tasks. 


Mental  model s.  It  Is  Important  to  determine  the  kinds  of  mental  structures  that  Influence 
performance.  How  can  these  be  conceptualized  and  Identified?  A  representative  book  by  Gentner 


26 


and  Stephens  (1983)  covers  a  wide  variety  of  studies  Including  mental  models  In  understanding 
electricity  and  physical  systems. 

Automatization.  What  conditions  of  instruction  favor  the  development  of  automaticlty  In 
basic  skills?  How  Is  this  related  to  the  development  of  higher  levels  of  job  performance? 
Representative  studies  Include  Perfetti  and  Lesgold  (1979),  on  the  interaction  of  basic  and 
higher-level  skills  and  verbal  comprehension;  Schneider  (1982),  on  training  for  automaticlty;  and 
Shiffrln  and  Schneider  (1977),  on  general  theory. 

Procedural Izatl on.  Experts  develop  not  only  declarative  knowledge  but  declarative  knowleage 
that  Is  attached  to  conditions  of  its  use  so  that  knowledge  becomes  effectively  useable. 
Representative  studies  include  Anderson  (1982),  an  article  on  theory;  Chi,  Glaser,  and  Rees 
(1982);  and  Lesgold  (in  press). 

Metacognition  and  learning  skill.  How  are  self-regulatory  and  self-management  skills  that 
Influence  learning  and  problem  solving  Influenced  through  training?  Representative  studies 
include  Brown  (1978),  on  theory  and  methodology;  Brown  and  Palincsar  (1982);  and  Gitomer  and 
Glaser  (1983). 

Training.  In  this  section  of  the  present  paper,  I  have  concentrated  on  automaticlty,  and  the 
references  given  under  the  heading  of  automatization  are  relevant  here. 

Problem  solving  and  knowledge.  How  is  knowledge  acquired  so  that  it  is  effectively  used  as  a 
basis  of  learning  and  problem  solving?  Representative  studies  include  Collins  and  Stevens 
(1982);  and  Glaser  (1983). 

Naive  and  pedagogical  theories.  How  can  the  naive  theories  of  a  learner  be  determined,  and 
how  can  they  be  used  as  a  basis  for  Instruction?  A  representative  article  is  by  McCloskey, 
Caramazza,  and  Green  (1980). 

Theory  and  practice  in  training.  In  this  section,  I  refer  to  the  importance  of  determining 
the  Intelligent  “bugs"  of  the  learner  that  can  be  used  as  a  basis  for  training.  Representative 
articles  Include  Brown  and  Burton  (1978);  and  Glaser  (1981).  Also  In  this  section,  I  refer  to 
the  macrovariables  of  teaching  and  classroom  management.  A  representative  article  by  Glaser 
(1982)  has  a  section  on  school  processes  with  a  variety  of  references. 
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CHAPTER  5 


CONTRIBUTIONS  OF  COGNITIVE  PSYCHOLOGY  TO 
LEARNING  FROM  INSTRUCTION  IN  AIR  FORCE  TRAINING 

Thomas  J.  Shuell 

State  University  of  New  York  at  Buffalo 


During  the  past  10  or  15  years,  there  has  been  a  tremendous  shift  In  the  way  psychologists 
think  about  Issues  related  to  learning  and  training.  This  change  In  perspective  has  been  heavily 
influenced  by  two  separate  but  related  revolutions:  the  computer  revolution  with  its  emphasis  on 
Information  processing  and  the  cognitive  revolution  with  Its  emphasis  on  mental  activities.  The 
Impacts  of  both  have  been  widespread.  On  topics  closely  associated  with  training,  for  example, 
the  influence  can  be  seen  clearly  In  current  thinking  about  the  nature  of  mental  abilities  and 
psychometric  tests,  as  well  as  In  current  thinking  about  the  nature  of  human  learning. 

This  paper  will  focus  primarily  on  the  major  themes  evident  In  modern-day  cognitive 
psychology  and  the  Influence  they  have  had  on  the  way  we  currently  think  about  human  learning  and 
Instruction.  At  present,  for  example,  learning  Is  generally  considered  to  be  an  active, 
constructive  process  rather  than  a  passive,  reproductive  process  (as  was  the  general  case  10  to 
20  years  ago).  The  emphasis  Is  no  longer  strictly  on  behavior;  rather.  It  Is  on  the  mental 
processes  and  knowledge  structures  that  can  be  Inferred  from  behavioral  Indices.  This  change  in 
thinking  about  the  nature  of  learning  Is  beginning  to  produce  corresponding  changes  In  the  way  we 
think  about  Instruction  and  training.  There  Is  Increasing  emphasis,  for  example,  on  the  way 
specific  content  and  Instructional  procedures  engage  and  elicit  the  psychological  processes  and 
knowledge  structures  appropriate  for  the  desired  learning  outcome  to  be  achieved. 

These  various  changes  In  the  way  psychologists  think  about  learning,  and  Its  relationship  to 
instruction,  need  to  be  considered  in  establishing  a  research  agenda  for  a  learning  laboratory  of 
the  type  being  discussed  here.  The  purpose  of  research  conducted  In  such  a  facility  is  to 
provide  a  theoretical  and  empirical  knowledge  base  for  improving  the  effectiveness  of  Air  Force 
training,  both  In  the  more  formal  schools  and  In  the  less  structured  training  that  occurs  "on  the 
job."  If  we  are  to  ask  the  best  and  most  appropriate  research  questions,  then  we  need  to  have 
some  Idea  of  the  various  factors  that  Influence  learning  In  the  different  training  environments 
found  in  the  Air  Force,  how  these  factors  are  Interrelated,  and  how  it  might  be  possible  to 
Integrate  them  In  a  meaningful  way.  The  next  section  of  this  paper  will  discuss  some  of  these 
general  Issues  In  more  detail.  Subsequent  sections  will  discuss  specific  ways  in  which  cognitive 
psychology  has  Influenced  current  research  on  learning  and  some  of  the  specific  research  Issues 
that  might  be  considered  In  an  Air  Force  learning  research  laboratory. 


Learning  from  Instruction 


Two  characteristics  of  traditional  research  on  learning  and  cognition  -  especially  the 
better,  more  rigorous  scientific  research  -  seriously  limit  the  usefulness  of  this  research  when 
It  comes  to  solving  practical  problems  such  as  those  involved  In  trying  to  improve  the 
effectiveness  of  Air  Force  training.  The  first  of  these  Is  the  lack  of  an  explicit  concern  for 
Instruction  and  application.  The  second  is  that  most  of  this  research  (as  is  the  case  with  most 
basic,  scientific  research)  is  descriptive  in  nature  (l.e.,  the  goal  of  this  research  is  to 
identify  and  describe  the  various  factors  that  Influence  the  learning  process,  with  very  little. 
If  any,  thought  given  toward  how  the  findings  of  this  research  relate  to  the  issues  Involved  in 
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using  It  to  Improve  training  outcomes).  That  Is,  the  data  and  conclusions  produced  by  this 
traditional  type  of  research  are  not  completely  appropriate  for  the  task  at  hand.  At  times.  It 
1$  almost  as  If  we  have  a  good  answer,  but  the  answer  Is  to  a  different  question  than  the  one  In 
which  we  are  really  most  Interested. 

Most  Individuals  concerned  with  applying  scientific  knowledge  to  the  solution  of  practical 
problems  have  envisioned  a  linear  model  In  which  basic  research  findings  move  through  a  series  of 
stages  or  steps  toward  ultimate  Implementation  and  dissemination  In  a  more-or-less  linear 
sequence  (e.g.,  Hllgard,  1964).  I  have  become  thoroughly  convinced,  however,  that  this  linear 
model  Is  totally  Inappropriate  for  any  kind  of  meaningful  research  or  development  with  regard  to 
practical  problems  (Glaser,  1973;  Shuell,  1982).  We  need  to  stand  the  whole  research  and 
development  process  on  Its  head  and  realize  that  basic  research  In  psychology  Is  a  descriptive 
science  whereas  what  we  need  for  Improving  training  and  education  Is  a  prescriptive  science.  The 
nature  of  a  prescriptive  science  very  different  from  a  descriptive  science,  and  these 
differences  should  be  taken  Into  account  when  planning  for  research  on  learning  from  Instruction 
(Shuell,  1981b,  1982). 


Descriptive  vs.  Prescriptive  Science 

Science,  as  we  normally  think  of  It,  Is  concerned  with  understanding  the  nature  of  naturally 
occurring  phenomena.  This  understanding  typically  takes  the  form  of  an  accurate  (l.e.,  reliable 
and  valid)  description  of  the  phenomenon  being  Investigated.  Typically,  It  Is  assumed  that  there 
Is  a  single  best  answer  or  description  of  the  phenomenon;  all  we  have  to  do  Is  discover  It.  For 
purposes  of  a  descriptive  science,  for  example.  It  Is  sufficient  to  show  that  the  way  In  which  an 
Individual  represents  a  problem  determines  how  successful  he  or  she  will  be  In  finding  a 
successful  solution  to  the  problem.  While  various  Issues,  such  as  the  conditions  necessary  for 
obtaining  the  effect,  the  extent  to  which  various  factors  may  Influence  the  likelihood  of 
obtaining  the  effect,  and  so  forth,  may  be  investigated,  the  purpose  of  the  research  is  to 
describe  the  relationship  by  determining  the  extent  and  conditions  under  which  the  phenomenon 
occurs. 

For  Instructional  purposes,  however,  this  description  Is  Inadequate.  Instead,  what  we  need 
Is  empirical  evidence  on  the  best  way  for  a  student  to  represent  a  problem  In  a  given  situation 
and  on  what  techniques  or  methods  an  Instructor  could  use  to  help  him  or  her  represent  the 
problem  In  the  most  appropriate  and  efficient  manner.  A  prescriptive  science.  In  contrast  to  a 
descriptive  science,  recognizes  that  there  can  be  more  than  one  scientifically  valid  way  of 
accomplishing  the  same  end  and  seeks  to  provide  theoretical  models  and  empirical  data  directly 
relevant  to  the  task  faced  by  the  individual  who  must  make  decisions  about  what  to  do  in  a 
particular  situation.  For  the  most  part,  an  Integrated  and  systematic  body  of  scientific 
knowledge  appropriate  for  this  type  of  practical  decision  making  simply  is  not  available  at  the 
present  time. 

The  chief  purpose  of  a  prescriptive  science  Is  to  create  artifacts  that  will  serve  some 
purpose  (Simon,  1981).  That  Is,  things  that  do  not  exist  naturally  In  nature  are  created  or 
Invented  In  order  to  accomplish  something  we  want  done.  For  example,  a  water  wheel  is  built 
(nothing  occurs  In  quite  that  form  In  nature)  In  order  to  turn  a  grain  mill  or  an  electric 
generator.  Similarly,  an  Instructor  may  decide  to  lecture  on  a  particular  topic  in  order  to  have 
trainees  learn  certain  information.  Usually  there  is  more  than  one  way  in  which  such  a  purpose 
can  be  accomplished;  the  grain  mill  can  also  be  turned  by  oxen  walking  in  a  circle,  and  the 
Information  to  be  learned  In  class  may  be  equally  well  acquired  by  having  the  trainees  work  on  a 
mockup  or  by  discussing  a  specified  problem  In  small  groups. 
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While  soae  Methods  for  accomplishing  a  particular  purpose  may  be  better  than  others, 
frequently  the  criteria  Involved  In  Identifying  the  "better*  or  "best"  approach  are  complex,  with 
a  variety  of  tradeoffs  being  Involved.  Often  the  only  thing  that  can  be  said  Is  that  the  various 
approaches  represent  different  ways  of  going  about  accomplishing  the  task,  each  with  different 
consequences  and  side  effects  (Shuell,  1982).  Research  on  human  learning  with  the  overall 
purpose  of  ultimately  Improving  the  effectiveness  of  training  or  Instruction  should  be  designed 
with  these  various  factors  explicitly  In  mind.  One  way  to  accomplish  this  goal  1$  to  develop  a 
conceptual  model  that  can  assist  In  guiding  both  basic  and  more  applied  research  on  learning  from 
Instruction. 


Toward  a  Prescriptive  Model  of  Learning  from  Instruction 


Recently,  I  have  developed  a  model  that  Incorporates  these  considerations  (Shuell,  1981b). 
The  model  Is  based  on  the  notion  that  In  order  to  make  useful  Instructional  decisions  based  on 
valid  scientific  findings,  we  need  to  develop  a  knowledge  base  that  recognizes  from  the  very 
beginning  that  there  are  two  types  of  constraints  or  "givens"  whenever  one  makes  an  Instructional 
decision. 

The  first  constraint  or  given  specified  by  the  model  Is  the  nature  of  the  outcome.  What 
exactly  are  we  trying  to  help  the  students  achieve?  Is  It  a  cognitive  goal,  an  affective  goal, 
the  removal  of  a  perceptual  problem  or  misconception,  or  something  else?  Various  types  of 
knowledge  characteristic  of  particular  Instructional  outcomes  can  be  Identified,  and  some  of 
these  different  types  of  knowledge  will  be  discussed  In  more  detail  later  In  this  paper.  The 
concern  for  the  nature  of  the  desired  outcome  extends  beyond  the  mere  specification  of 
Instructional  objectives.  The  model  emphasizes  the  Importance  of  Identifying  the  nature  of  the 
Instructional  goal  In  a  psychological  manner  that  Is  consistent  with  (a)  the  other  given 
specified  by  the  model  (to  be  discussed  next)  and  (b)  the  selection  of  an  Instructional 
Intervention  that  appropriately  matches  the  two  givens  and  that  has  a  high  probability  of  leading 
to  the  efficient  attainment  of  the  goal.  It  1$  likely  that  many  or  most  of  these  outcomes  can 
come  from  existing  analyses  and  descriptions  of  Air  Force  jobs  and  job  composites,  (Gagne's  paper 
previously  discussed  these  types  of  outcomes). 

The  second  given  Identified  by  the  model  has  to  do  with  the  nature  of  the  existing 
situation.  When  making  decisions  about  the  most  appropriate  type  of  training  to  provide,  one  Is 
faced  with  the  realities  of  the  situation  In  which  the  training  must  be  accomplished,  and  these 
realities  place  constraints  on  the  type  of  Instruction  that  can  provide  the  most  effective 
training,  given  that  situation.  Included  In  this  component  of  the  model  are  many  more 
considerations  than  the  traditional  concern  for  the  prior  knowledge  of  the  learner.  These 
additional  factors  Include  concerns  for  the  resources  one  has  available,  concerns  for  the  social 
dynamics  that  exist  among  the  trainees  and  between  the  trainees  and  the  Intructor,  and  so  forth. 

Now,  given  these  two  types  of  constraints,  what  knowledge  about  the  psychology  of  learning 
from  instruction  does  an  Individual  need  to  have  In  order  to  make  a  viable  Instructional  decision 
based  on  the  best  scientific  evidence  available?  When  faced  with  making  a  decision  about 
training,  one  might  ask:  What  am  I  going  to  teach  next;  and  given  that  I  am  going  to  teach  this 
material  or  skill  next,  how  am  I  going  to  teach  it  In  order  for  learning  to  be  most  effective? 
In  making  these  Instructional  decisions,  one  Is  faced  with  teaching  particular  outcomes  within  an 
existing  situation,  and  those  making  the  decision  need  to  know  something  about  alternative 
procedures  that  will  help  the  students  to  get  from  Point  A  to  Point  B.  Thus,  a  means-end 
analysis  needs  to  be  performed,  and  a  viable  theory  of  Instruction  -  or  systematic  body  of 
psychological  knowledge  -  must  be  able  to  provide  scientific  data  relevant  to  this  task. 
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Unfortunately,  most  of  the  scientific  knowledge  currently  available  Is  not  organized  In  a 
■anner  that  lends  Itself  to  Making  these  types  of  practical  decisions.  What  Is  needed  Is 
research  that  can  say:  Given  these  outcomes,  .d  given  this  situation,  these  Halted  number  of 
possibilities  exist  for  effective  Instruction.  From  that  saall  pool  of  alternative  procedures,  a 
person  can  then  aake  a  reasonable  professional  judgment.  We  probably  will  never  reach  the  point 
where  we  are  capable  of  prescribing  specific  Instructional  procedures,  but  I  do  think  there  Is  a 
possibility  of  developing  a  body  of  knowledge  that  will  help  us  to  narrow  the  alternatives  In  a 
scientifically  valid  manner.  In  order  to  accomplish  this  goal,  however,  we  must  set  out  to  do 
research  conceptualized  in  a  manner  appropriate  for  the  task. 

In  discussing  research  on  learning  from  Instruction,  soMe  Mention  must  be  made  of  the 
findings  generated  from  classroom  research,  especially  the  process-product  research  (e.g., 
Brophy,  1979;  Brophy  4  Good,  In  press).  Until  recently,  classroom  research  and  learning  research 
have  been  Isolated  from  one  another.  Although  virtually  none  of  the  classroom  research  has  been 
conducted  In  a  context  similar  to  the  one  encountered  In  Air  Force  training,  many  of  the  findings 
can  undoubtedly  be  generalized  In  some  way  to  Air  Force  training  environments.  In  any  case, 
truly  productive  research  on  learning  from  Instruction  must  be  performed  with  an  awareness  of  the 
research  being  performed  on  classroom  learning,  and  research  performed  at  the  Air  Force  learning 
research  laboratory  should  reflect  this  awareness. 


Contributions  of  Cognitive  Psychol o 


Cognitive  psychology  has  brought  a  number  of  Identifiable  concerns  or  themes  to  the  field  of 
psychology  In  general,  and  educational  psychology  In  specific.  These  concerns  have  emphasized 
new  aspects  of  human  Information  processing  and  learning  that  need  to  be  Incorporated  Into  any 
attempt  to  understand  the  nature  of  effective  training  and  learning  from  Instruction.  Some  of 
the  topics  emphasized  by  modern-day  cognitive  psychology  Include  a  concern  for  understanding  what 
one  Is  learning  (l.e.,  comprehension),  knowledge  representation,  problem  solving,  and  the  nature 
of  Individual  differences  In  learning  from  Instruction.  These  topics  will  be  discussed  next,  and 
an  attempt  will  be  made  to  relate  the  concerns  represented  by  these  topics  to  research  that  could 
be  conducted  In  the  proposed  laboratory. 


Comprehension 

Probably  the  first  and  most  Important  of  the  Issues  emphasized  by  cognitive  psychology  Is  a 
concern  for  meaning  -  that  Is,  the  manner  In  which  an  Individual  extracts  meaning  from  some 
experience.  The  emphasis  here  Is  on  understanding,  rather  than  merely  on  learning  (or  learning 
how  to  perform)  the  material  or  task  being  acquired.  This  emphasis  Includes  a  concern  for 
comprehension  -  specifically,  or  at  least  most  extensively,  for  prose  comprehension. 


There  has  been  a  great  deal  of  research  on  written  comprehension  (e.g.,  Reder,  1980;  Spiro, 
Bruce,  &  Brewer,  1980).  Very  little  work,  however,  has  been  done  on  oral  comprehension,  which  is 
equally  important,  especially  since  we  convey  as  much  of  our  instruction  via  the  oral  modality 
(l.e.,  lectures,  etc.)  as  we  do  by  the  written  modality  (l.e.,  textbook,  handouts,  etc.).  A  few 
studies  (e.g.,  Anderson  i  Biddle,  1975;  Smiley,  Oakley,  Worthen,  Camp  lone,  i  Brown,  1977)  have 
compared  comprehension  In  the  two  modalities  without  finding  any  large  differences.  Researchers 
in  this  area  frequently  make  the  assumption  that  the  same  psychological  mechanisms  are 
responsible  for  both  written  and  oral  comprehension.  While  this  assumption  does  not  seem  totally 
unreasonable.  It  Is  an  Issue  that  we  really  have  not  pursued  very  far.  Given  Its  Importance  for 
training  and  learning  from  Instruction  In  a  variety  of  situations  (including  on-the-job 


training),  I  think  It  Is  worth  further  Investigation.  That  Is,  what  Is  the  nature  of  oral 
coaprehenslon,  as  well  as  the  nature  of  written  comprehension;  and  what  Is  the  role  of  each  In 
learning  from  Instruction  In  a  training  environment? 

A  third  aspect  to  this  question  of  comprehension  Is  a  concern  for  visual  comprehension.  This 
modality  Is  one  about  which  we  know  very  little  with  regard  to  what  constitutes  coaprehenslon, 
outside  of  the  traditional  work  on  spatial  abilities,  which  is  not  extremely  relevant  to  the 
concerns  being  discussed  here.  There  are  some  very  Interesting  Indications,  however,  that  visual 
comprehension  may  be  a  useful  topic  to  represent  knowledge  for  Instructional  purposes,  which  Is 
the  larger  concern  In  this  context.  It  may  be,  for  example,  that  In  some  cases  we  could 
represent  the  knowledge  that  a  trainee  needs  to  acquire  In  order  to  learn  specific  skills  In 
spatial  formats  that  would  lead  to  more  rapid  acquisition  than  would  be  the  case  If  we  presented 
the  Instruction  In  the  written  or  oral  modalities.  There  Is  some  evidence  (e.g..  Levin,  1981) 
which  shows  that  visual  mnemonics  can  be  very  powerful  Instructional  tools.  There  is  also  work 
by  Lynn  Cooper  (1982)  at  the  Learning  Research  and  Development  Center  (LRDC)  at  the  University  of 
Pittsburgh  which  Identifies  different  types  of  visual  processing  strategies  that  Individuals  use 
and  some  Indication  of  what  occurs  when  there  Is  a  match  or  mismatch  with  regard  to  Instructional 
presentations  -  that  Is,  what  happens  If  you  present  Instruction  that  Is  consistent  with  an 

Individual's  preferred  perceptual  processing  strategy  and  what  happens  If  you  do  not. 

From  an  Instructional  point  of  view,  there  has  been  a  fair  amount  of  research  on  aids  to 

comprehension  -  things  like  advanced  organizers,  adjunct  questions,  and  so  forth.  Rich  Mayer  (In 
press)  recently  has  completed  a  review  of  the  literature  on  comprehension  aids,  and  a  recent 
paper  by  Merl  Wlttrock  (1983)  discusses  many  of  the  same  Issues  from  a  slightly  different 
perspective.  In  addition,  the  visual  mnemonics  discussed  In  the  preceding  paragraph  can  also  be 
regarded  as  one  type  of  comprehension  aid.  The  concern  for  which  Instructional  aids  might 
facilitate  learning  and  comprehension  In  the  typ«s  of  training  environments  encountered  In  the 
Air  Force  Is  something  that  should  be  looked  at  In  terms  of  the  various  modalities  of 

comprehension  discussed  above,  for  such  aids  are  an  instructional  tool  that  deserves  further 
investigation. 

Types  of  Knowledge 

One  characteristic  of  modern-day  cognitive  psychology  Is  a  recognition  that  there  Is  more 
than  one  type  of  knowledge  and  that  a  concern  for  different  types  of  outcomes  should  be 
considered  In  designing  appropriate  learning/training  environments.  Although  this  realization  Is 
not  actually  a  new  thrust  In  learning  research.  It  has  taken  on  a  rather  different  flavor  since 
Bob  Gagne's  (e.g.,  1962,  1965)  seminal  work  20  years  ago  on  different  types  of  learning  and  the 
Implications  of  this  distinction  for  Instruction  (l.e.,  the  conditions  for  learning  and  the 
Implications  of  that  orientation  for  Instruction). 

Cognitive  psychology  acknowledges  that  there  are  different  types  of  knowledge,  most 
frequently  distinguishing  between  procedural  knowledge  and  propositional  knowledge  (also  referred 
to  as  declarative  knowledge  or  semantic  knowledge).  Procedural  knowledge  refers  to  one's 
knowledge  of  how  to  do  the  various  procedures  necessary  to  perform  some  task.  The  task  can  be 
either  psychomotor  or  intellectual,  and  Gagn£  (1977)  also  identifies  motor  skills  as  a  separate 
type  of  learned  outcome.  Propositional  knowledge,  on  the  other  hand,  refers  to  the  systematic 
and  organized  body  of  knowledge  that  we  have  about  something.  There  are  undoubtedly  other  types 
of  knowledge  as  well.  For  example,  Gagn4  and  White  (1978)  Identify  four  types  of  organized 
memory  structures:  (a)  Intellectual  skills,  (b)  networks  of  propositions  (propositional 
knowledge),  (c)  Images,  and  (d)  episodes  (autobiographical  and  temporal  data  information). 
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There  are  at  least  two  reasons  why  we  should  be  more  concerned  with  the  various  types  of 
outcomes  that  we  are  trying  to  achieve  In  training.  First,  the  different  outcomes  are  acquired 
most  efficiently  through  the  presence  of  different  learning  conditions;  consequently,  different 
types  of  Instructional  learning  environments  are  required,  depending  on  the  type  of  desired 
outcome  (Gagn4,  1965,  1977).  We  should  not  help  a  trainee  to  learn  a  procedure  In  the  same  way 
that  we  help  him  or  her  to  learn  a  proposition.  Second,  since  there  are  qualitatively  different 
types  of  outcomes,  we  cannot  expect  a  person  to  know  how  to  do  something  (procedural  knowledge) 
just  because  we  have  taught  him  or  her  something  about  It  (propositional  knowledge),  any  more 
than  we  can  expect  a  trainee  to  know  various  radio  call  signs  after  teaching  him  or  her  the  names 
of  various  types  of  radio  equipment.  In  order  to  provide  the  most  effective  training  possible, 
we  need  to  know  both  the  type  of  outcome  we  are  seeking  and  the  training  conditions  best  suited 
for  achieving  that  type  of  outcome. 

How  do  we  go  about  this?  There  are  several  lines  of  research  that  might  be  pursued  by  the 
learning  laboratory.  One  thing  that  has  Impressed  me  about  some  of  the  job  descriptions  and 
objectives  currently  being  used  In  the  Air  Force  -  although  It  Is  possible  that  I  do  not  fully 
understand  the  system  -  Is  that  there  does  not  seem  to  be  any  way  of  identifying  similar  types  of 
objectives  In  terms  of  psychological  factors  that  could  be  related  to  different  types  of 
Instructional  modes.  The  objectives  seem  to  be  primarily  descriptive  of  the  desired  outcomes, 
rather  than  having  been  analyzed  or  Identlfed  In  terms  of  the  cognitive  processes  or  mental 

activities  that  need  to  be  engaged  or  are  necessary  In  order  for  an  Individual  to  perform  them. 

Although  I  am  not  a  fond  lover  of  taxonomies,  perhaps  there  should  be  some  concern  for  the  types 
of  knowledge  that  we  are  trying  to  help  the  trainees  achieve  with  regard  to  both  on-the-job  and 
end-of-tral nl ng  performance.  We  need  to  know  more  about  the  type  of  categories  that  are  most 
meaningful  and  useful  for  Air  Force  training.  Then  we  need  to  know  more  about  the  types  of 
training  that  are  most  appropriate  for  acquiring  the  various  types  of  outcomes.  Do  we,  for 
example,  teach  procedures  differently  from  the  way  we  teach  declarative  knowledge?  Are  concepts 
taught  differently  than  the  paired-associate  learning  that  one  engages  In  when  trying  to  achieve 
certain,  necessary  terminology?  Research  on  both  of  these  Issues  could  be  pursued  at  the 
learning  laboratory. 

A  related  concern  that  has  been  receiving  Increased  attention  In  recent  years  Is  the  attempt 

to  describe  and  understand  the  nature  of  competence  as  reflected  In  differences  between  experts 

and  novices  In  a  particular  field.  (This  concern  was  further  expanded  and  related  more  directly 
to  Air  Force  training  In  Glaser's  paper.)  The  focus  of  this  research  Is  on  qualitative 
differences  (e.g.,  differences  In  learning  strategies  employed  or  differences  in  the  way  they 
approach  problems)  rather  than  quantitative  differences  (l.e.,  mere  differences  In  the  amount  of 
knowledge  possessed).  Bob  Glaser  and  others  at  LRDC  (e.g.,  Chi,  Glaser,  A  Rees,  1982;  Voss, 
Tyler,  A  Yengo,  1983)  have  performed  much  of  this  research,  although  other  Investigators  have 
also  been  active  In  the  field  (e.g.,  Larkin,  McDermott,  Simon,  A  Simon,  1980).  This  description 
of  differences  between  experts  and  novices  in  a  field  may  be  a  useful  way  for  going  about  the 
task  of  meaningfully  describing  the  type  of  knowledge  or  competence  that  we  want  various  trainees 
to  acquire.  For  example,  what  does  a  radio  operator  do  In  carrying  out  his  or  her  job?  What  are 
the  strategies  that  are  used  In  performing  this  job  in  a  competent  manner?  How  do  these 
strategies  differ  from  those  used  by  a  novice  and  can  they  be  specified  In  a  manner  relevant  to 
the  task  of  moving  a  person  from  being  a  novice  to  an  expert  in  the  way  the  job  Is  performed? 
How  we  actually  go  about  doing  this  Is  not  completely  clear,  but  one  of  the  problems  that  might 
be  addressed  at  the  laboratory  would  be  an  attempt  to  apply  some  of  the  more  qualitative 
approaches  that  have  been  used  In  defining  the  nature  of  competence  to  on-the-job  performance 
being  carried  on  by  Air  Force  personnel  In  the  field. 

We  also  should  be  aware  of  the  fact  that  the  nature  of  the  knowledge  being  acquired  may 
change  as  learning  progresses;  as  one  moves  through  a  unit  or  set  of  procedures  for  learning 
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something,  different  types  of  Instructional  support  may  be  needed.  John  Anderson  (1982)  has 
recently  described  the  various  stages  one  goes  through  In  learning  a  procedure  or  procedural 
knowledge.  He  has  suggested  three  stages  that  parallel  Fitts'  (1964)  stages  of  learning  a  motor 
skill,  although  Anderson  refers  to  the  newer  stages  by  somewhat  different  names. 

The  first  of  these  stages  Anderson  refers  to  as  the  declarative  stage  In  which  the  concern 
for  declarative  knowledge  Is  paramount.  It  may  be  that  In  this  stage  we  need  a  fair  amount  of 
Instructional  support  In  terms  of  mnemonics  or  something  that  would  aid  the  learner  In  holding 
things  together  long  enough  for  the  core  material  to  be  Integrated  Into  a  meaningful  part  of  his 
or  her  knowledge  structure.  The  second  stage  that  Anderson  has  identified  Is  called  knowledge 
compilation.  This  stage  Is  concerned  with  the  transition  from  knowledge  represented  In  a 
declarative  form  to  knowledge  represented  In  a  procedural  form  and  Is  Interpretive  and 
Integrative  In  nature.  Finally,  the  stage  In  which  the  skill  has  become  autonomous  and  there  Is 
a  gradual  Increase  In  the  speed  with  which  the  skill  can  be  performed  Is  referred  to  by  Anderson 
(1982)  as  the  procedural  stage. 

If  our  goal  Is  to  train  people  to  perform  procedures,  which  seems  to  be  a  large  part  of  the 
technical  training  conducted  by  the  Air  Force,  we  may  need  to  become  more  concerned  for  the  type 
of  declarative  knowledge  we  need  to  use,  ensuring  either  Its  usefulness  as  a  crutch  for  Initial 
learning  or  Its  relevance  to  the  task  In  a  more  direct  manner.  That  Is,  what  do  we  need  to  teach 
over  and  above  the  specific  procedures  and  what  Is  needed  to  support  the  transitions  between  the 
various  stages  of  learning? 


Individual  Differences 

A  third  concern  does  not  strictly  grow  out  of  cognitive  psychology,  but  It  Is  so  much  at  the 
core  of  current  thinking  about  learning  from  Instruction  that  It  would  be  an  oversight  not  to 
mention  It.  This  concern  is  for  the  nature  of  individual  differences  in  learning  (which  has 
taken  on  a  decidedly  cognitive  flavor)  and  a  corresponding  concern  for  aptitude- treatment 
Interactions  (ATIs).  ATIs  pertain  to  situations  In  which  an  instructional  method  that  is  optimal 
for  one  type  of  student  may  produce  poorer  learning  for  a  different  type  of  student,  with  the 
reverse  being  true  for  a  different  Instructional  method  (Cronbach  A  Snow,  1977).  While  ATI 
research  has  not  been  overwhelming  In  terms  of  Its  findings,  some  fairly  fundamental  conclusions 
can  be  stated  with  confidence.  For  example,  low  ability  students  seem  to  do  consistently  better 
in  a  training  environment  that  has  a  high  degree  of  structure,  thereby  placing  a  low 
informational  processing  load  on  the  learner.  High  ability  students,  on  the  other  hand,  perform 
at  a  slightly  lower  level  In  that  kind  of  an  environment  than  they  do  In  an  Instructional 
environment  that  places  a  high  Information  processing  load  on  the  learner  (Snow,  1977). 

Put  in  absolute  terms,  It  Is  not  clear  where  Air  Force  trainees  would  be  with  regard  to 
ability.  Neither  Is  it  clear  what  the  effects  would  be  if  one  looked  at  prior  knowledge  (e.g., 
Tobias,  1976)  rather  than  ability.  Findings  of  this  nature,  however,  may  be  worth  pursuing. 
Other  than  that,  the  ATI  research  has  not  been  all  that  fruitful.  I  think  there  are  two  reasons 
why  this  might  be  the  case.  First,  I  think  that  not  enough  consideration  has  been  given  to  the 
various  types  of  knowledge  involved  in  the  desired  outcomes;  that  is,  TOIs  (treatment-outcome 
Interactions)  and  AOIs  (aptitude-outcome  Interactions)  may  be  Involved  as  well  as  ATIs. 

The  other  reason  has  to  do  with  the  fact  that  to  a  very  large  extent,  ATI  research  has  used 
off-the-shelf  tests  to  measure  aptitudes;  tests  that  were  designed  for  other  purposes  -  namely, 
unidimensional  prediction  in  a  more-or-less  fixed  instructional  environment.  Perhaps  what  we 
need  to  do,  if  we  are  serious  about  trying  to  develop  a  viable  ATI  technology,  is  to  pursue  the 
possibility  that  we  need  to  develop  completely  new  aptitude  tests  designed  for  differential 
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prediction.  The  notion  of  new  kinds  of  aptitudes  obviously  Is  not  a  totally  fresh  Idea  (Glaser, 
1972).  What  I  have  In  nlnd  here  Is  to  go  through  the  process  of  developing  tests  whose  purpose 
would  be  one  of  differential  prediction  for  different  learning  environments  and/or  Instructional 
procedures.  That  Is,  you  take  different  types  of  outcomes  or  different  types  of  Instructional 
treatments,  develop  a  pool  of  test  Items,  and  then  select  Items  for  the  test  that  will 

discriminate  appropriately.  Thus,  we  would  actually  build  new  psychometric  instruments  for  the 
purpose  of  differential  prediction  rather  than  unilateral  prediction,  which  Is  the  purpose  of 
traditional  psychometric  tests. 

There  are  several  ways  one  might  go  about  this  task.  Ideally,  one  needs  a  theory  of  what  Is 
and  Is  not  Important  In  these  different  Instructional  environments  and  the  types  of  mental 

activities  that  students  engage  In  when  learning  a  specific  type  of  outcome  In  a  specific  type  of 
learning  environment.  It  would  help  to  have  a  viable  theory  of  cognition  to  guide  this 
development,  and  I  think  we  may  have  the  beginnings  of  what  might  prove  to  be  appropriate 
conceptions  of  (a)  a  useful  cognitive  model  of  learning,  (b)  the  task  demands  of  various 
Instructional  environments,  and  (c)  the  task  demands  of  various  outcomes.  For  example, 
considerable  progress  has  been  made  In  recent  years  In  developing  a  process  theory  of  aptitudes 
(e.g..  Snow,  1980).  This  work  Is  very  encouraging  In  terms  of  developing  a  better  understanding 

of  aptitudes  and  how  they  relate  to  Instruction.  In  general,  however,  these  aptitudes  tend  to  be 

of  a  general  nature  In  the  sense  of  cutting  across  various  subject-matter  domains  and 
environmental  (Instructional)  task  demands. 

There  have  been  some  notable  and  Important  attempts  In  the  past  5  years  to  understand  the 
nature  of  psychometric  performance.  The  concern  of  this  research  has  been  for  Identifying  the 
cognitive  processes  responsible  for  the  performance  observed  on  psychometric  tests  -  that  Is,  as 
Carroll  (1976)  has  so  aptly  put  It,  concern  for  "psychometric  test  as  cognitive  tasks."  Two 
basically  different,  but  not  mutually  exclusive,  paradigms  have  been  used  In  this  research. 
Pellegrino  and  Glaser  (1979)  have  labeled  these  two  paradigms  the  "cognitive  components  approach" 
and  the  "cognitive  correlates  approach."  In  the  cognitive  components  approach,  performance  on 
parameters  of  complex  Information-processing  tasks  taken  directly  from  Items  on  psychometric 
tests  (e.g.,  analogy  Items)  are  correlated  with  scores  from  the  psychometric  tests,  on  the 
assumption  that  Individual  differences  on  the  component  tasks  underlie  Individual  differences  in 
overall  test  performance  (e.g.,  Sternberg,  1979).  In  the  cognitive  correlates  approach, 
parameters  from  the  simple  kinds  of  Information-processing  tasks  studied  In  the  laboratory  are 
correlated  with  scores  from  tests  of  mental  ability,  on  the  assumption  that  Individual 
differences  In  the  cognitive  processes  represented  by  these  tasks  underlie  observed  scores  on 
psychometric  tests  of  mental  abilities  (e.g..  Hunt,  Frost,  &  Lunneborg,  1973).  These  approaches 
could  also  be  used  to  Investigate  various  types  of  complex  learning  tasks,  such  as  those 
encountered  In  many  types  of  training  situations. 

In  another  sense,  because  we  are  concerned  primarily  with  performance  on  learning  tasks, 
psychometric  Instruments  to  a  very  large  extent  are  not  merely  attempts  to  understand  the 
Individual  differences  In  psychometric  ability  but  rather,  the  Individual  differences  In  learning 
tasks  themselves.  Some  of  my  own  research  (e.g.,  Shuell,  In  press)  and  some  of  the  research  of 
John  8ransford  (e.g.,  Bransford,  Stein,  Shelton,  &  ftrtngs,  1982;  Bransford,  et  al..  In  press)  has 
followed  this  approach.  We  start  by  taking  some  type  of  learning  task  (either  laboratory-type 
tasks,  which  at  this  early  stage  of  research  Is  not  totally  unreasonable,  or  more  complex  types 
of  tasks  similar  to  those  that  might  be  encountered  In  a  classroom)  as  the  criterion  task  we  are 
Interested  In  understanding.  Next  we  ask,  "Since  individuals  differ  In  their  ability  to  learn 
these  tasks.  Is  It  possible  to  understand  the  nature  of  this  differential  performance  through 
either  some  sort  of  a  componentlal  or  a  correlational  approach?" 
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In  sow  of  my  research  (e.g.,  Shuell,  In  press),  for  example,  we  have  looked  at  factors  that 
might  reduce  some  of  the  differences  In  performance  between  the  good  learners  and  the  poor,  slow 
learners  on  a  free-recall  task.  In  one  study,  we  used  categorized  word  lists  and  found  that 
Individuals  Identified  as  either  fast  learners  or  slow  learners  on  a  separate  free-recall  test  of 
unrelated  words  profited  differentially  froai  Instructions  to  organize  the  material  during 
learning.  These  instructions  Improved  the  performance  of  the  poor  learners  tremendously  but 
actually  had  a  small  negative  effect  on  the  performance  of  the  good  learners.  These  data  can  be 
used  In  two  ways.  First,  theoretically.  In  that  perhaps  one  of  the  things  differentiating  good 
and  poor  learners  Is  something  analogous  to  a  production  deficiency.  Slow  learners  may  have 
appropriate  strategies  but  do  not  use  them  unless  they  are  prompted.  Second,  the  data  also  can 
be  looked  at  from  a  more  practical  orientation;  l.e.,  this  may  be  a  useful  way  to  provide 
Instructional  support.  If  one  were  to  pursue  this  approach  In  the  present  context,  one  might 
take  jobs  performed  In  the  field,  look  at  the  learning  or  acquisition  of  those  jobs,  and  then  try 
to  Identify  those  factors  that  explain  Individual  differences  In  the  rate  of  learning  those  jobs 
and  how  sow  of  those  differences  might  be  reduced. 


Cognitive  Research  on  Learning 


Until  recently,  cognitive  psychology  has  shown  very  little  concern  for  learning  as  such  - 
that  Is,  concern  for  the  factors  or  variables  that  Influence  changes  In  human  performance, 
knowledge  structures  and/or  conceptions  (Anderson,  1982;  Greeno,  1980;  Langley  i  Simon,  1981). 
Rather,  cognitive  psychology  has  focused  on  Identifying  and  describing  the  various  stages  and 
processes  that  are  Involved  when  humans  process  Information.  The  most  significant  Impact  that 
cognitive  psychology  has  had  on  the  psychology  of  learning  Is  to  change  the  conception  of 
learning  from  a  process  that  Is  passive  and  reactive  to  a  process  that  Is  active  and  goal 
oriented.  Although  this  Is  an  extrewly  major  and  Important  Impact  In  Itself,  there  have  been 
few  attempts  so  far  to  follow  up  this  change  In  conception  with  Investigations  of  the  variables 
that  might  be  responsible  for  these  psychological  changes  we  call  learning. 

There  have  been,  however,  several  attempts  to  articulate  the  role  of  learning  from  a 
cognitive-psychology  and  human-information-processing  perspective.  Most  of  the  recent  work  has 
occurred  within  the  area  of  artificial  Intelligence,  where  the  goal  has  been  to  develop  computer 
programs  that  can  learn.  The  only  formal  definition  of  learning  from  this  perspective  that  I 
have  cow  across  Is  the  following  one  suggested  by  Langley  and  Simon  (1981): 

Learning  Is  any  process  that  modifies  a  system  so  as  to 
Improve,  more  or  less  Irreversibly,  Its  subsequent  performance 
of  the  same  task  or  of  tasks  drawn  from  the  same  population. 

(p.  367) 

In  many  ways,  the  differences  between  this  definition  and  more  traditional,  behavioristic 
definitions  of  learning  appear  to  Involve  flavor  more  than  substance,  although  some  very 
Important  differences  do  exist  between  the  two  orientations. 

Probably,  the  first  modern  attempt  to  develop  a  cognitive  conception  of  learning  was 
Ruwlhart  and  Norman's  (1978)  logical  analysis  of  the  nature  of  learning  within  a  schema-based 
representational  system.  Thinking  about  knowledge  representation  and  memory  in  terms  of  schemata 
Is  very  common  In  cognitive  psychology.  According  to  Rumelhart  and  Norman,  there  are  three 
qualitatively  different  kinds  of  learning. 

1.  Accretion,  or  the  encoding  of  new  information  In  terms  of  existing  schemata. 
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2.  Tuning  or  schema  evolution,  which  Involves  the  slow  Modification  and  reflnenent  of  a 
schema  as  a  result  of  using  It  In  different  situations. 

3.  Restructuring  or  schema  creation,  which  Is  the  process  whereby  new  schemata  are  created. 

Although  aost  models  of  memory  Involve  learning  by  means  of  accretion,  Rumelhart  and  Norman 
(1981)  go  on  In  a  more  recent  chapter  to  suggest  that  learning  which  Involves  the  creation  of  new 
scheaata  occurs  as  the  result  of  analogical  process;  that  is,  we  learn  new  schemata  by  relating 
new  Information  to  old  schemata  In  analogical  ways. 

John  Anderson  (1982)  has  suggested  that  we  learn  procedure  knowledge  by  making  Inferences 
from  declarative  (propositional)  knowledge.  He  has  developed  a  computer  program,  called  ACT, 
capable  of  learning  how  to  do  geometry  proofs.  Briefly,  the  program  Involves  an  adaptive 
production  system  that  engages  In  the  types  of  learning  referred'  to  above  as  tuning.  Three 
learning  mechanisms  are  used  as  the  basis  of  tuning: 

1.  Generalization,  a  process  by  which  production  rules  become  broader  In  their  range  of 
applicability. 

2.  D1 scrlml nation,  a  process  by  which  production  rules  become  narrower  In  their  range  of 
applicability. 

3.  Strengthening,  a  process  by  which  better  rules  are  strengthened  and  poorer  rules  are 
weakened. 

These  approaches  represent  the  beginning  of  a  concern  to  develop  a  viable  cognitive  theory  of 
learning.  Although  they  do  represent  a  major  advance,  they  still  seem  to  me  to  be  rather  removed 
from  instructional  Implications.  An  attempt  to  bring  together  the  concerns  of  traditional 
learning  psychology  with  the  newer  concerns  of  cognitive  psychology  Is  something  that  should  be 
at  the  very  focal  point  of  the  new  laboratory.  Considerable  thought  needs  to  be  given  to 
developing  the  most  productive  relationship  possible  between  learning  and  cognition,  on  the  one 
hand,  and  Instructional  procedures  appropriate  for  Air  Force  training,  on  the  other. 


Research  on  Learning  Variables  That  Have  Instructional  Relevance 

The  final  concern  that  will  be  discussed  here  Is  for  research  on  those  learning  variables 
that  we  know  have  Important  Instructional  Implications  or  that  have  effects  as  far  as  teaching  Is 
concerned.  Ue  know  a  number  of  things  that  are  worth  pursuing  further. 

If  we  accept  memory  and  transfer  as  the  bottom  line  of  any  training  or  educational  program 
(l.e.,  the  true  goal  of  training  Is  not  to  have  the  trainees  demonstrate  learning  in  the 
classroom,  but  to  have  them  retain  and  be  able  to  use  the  knowledge  they  have  acquired  In  novel 
situations),  then  we  need  to  consider  some  of  the  factors  that  Influence  these  goals.  (See 
Royer's  paper  for  a  further  discussion  of  transfer.)  Ue  know  quite  a  lot  about  how  to  improve 
both  memory  and  transfer  by  providing  for  the  appropriate  conditions  during  learning.  Ue  need  to 
extend  this  knowledge,  especially  In  the  prescriptive  sense  discussed  in  the  preceding  section, 
and  Incorporate  It  Into  our  training  programs. 

For  example,  we  know  that  distribution  of  practice  during  learning  has  extremely  large 
effects  In  facilitating  memory  (Bloom  &  Shuell,  1981;  Shuell,  1981a),  at  least  with  verbal 
materials  and  24-hour  distribution  Intervals.  Uill  these  same  procedures  work  for  more 
meaningful  materials?  Ue  have  tried  In  our  laboratory  to  extend  these  findings  to  both  concept 


learning  and  prose  learning,  but  so  far,  our  results  have  been  Inconclusive  for  a  variety  of 
reasons  that  are  Irrelevant  here. 


The  use  of  mnemonics  -  both  visual  and  verbal  -  Is  another  topic  for  which  there  are 
encouraging  results.  Are  there  situations  In  Air  Force  training  where  mnemonics  could  be  used 
appropriately,  either  as  a  crutch  on  the  way  toward  making  performance  automatic,  or  In  ways  that 
are  more  directly  related  to  the  task  being  acquired?  ty  hunch  Is  that  mnemonics  have  their  best 
effects  as  temporary  adjuncts  -  that  Is,  they  serve  like  a  glue  to  hold  things  together  until  the 
material  Is  Integrated  Into  a  larger  knowledge  structure. 


Possibilities  for  Research 


Research  conducted  In  an  Air  Force  learning  research  laboratory  can  and  should  take  a  variety 
of  different  forms  and  directions.  Specific  research  projects,  however,  should  Incorporate  the 
various  types  of  concerns  discussed  earlier  In  this  paper.  For  example,  research  focusing  only 
on  how  skills  are  learned  and/or  retained  Is  incomplete  as  far  as  the  overall  purpose  of  the 
laboratory  Is  concerned;  Instructional  variables  and  concern  for  how  one  might  Increase  the 
effectiveness  of  the  training  program  -  designed  to  produce  specific  learning  outcomes  -  should 
be  evident  In  each  project.  To  some  extent,  this  orientation  Involves  thinking  about  learning 
and  various  learning  variables  In  a  somewhat  different,  more  cognitive  manner  than  Is  the  case 
with  traditional  research  on  learning,  as  well  as  being  aware  of  the  necessary  relationship 
between  learning  and  Instructional  variables. 

This  section  of  the  paper  will  discuss  several  possible  research  projects  and  try  to  show  how 
the  various  concerns  voiced  above  might  be  Integrated  In  a  productive  manner.  The  specific 
examples  discussed,  of  course,  are  Intended  merely  to  be  representative  of  the  types  of  research 
that  might  be  pursued  at  the  laboratory. 


Acquisition  and  Retention  of  Procedural  Knowledge 


Host  learning  Involved  In  Air  Force  training  Is  procedural  In  nature  (l.e.,  using  some 

Intellectual  or  physical  skill  In  such  endeavors  as  operating,  maintaining,  or  repairing  a 

specific  piece  of  equipment),  although  some  declarative  (propositional)  knowledge  (e.g.,  learning 
radio  call  signs,  specific  regulations,  or  provisions  of  the  Uniform  Code  of  Military  Justice  Is 
also  seen  as  being  Important  In  Its  own  right.  Yet,  this  emphasis  on  the  performance  of 

procedural  tasks  does  not  necessarily  mean  that  the  learning  of  declarative  knowledge  Is  of 
little  Importance  In  Air  Force  training. 

Aside  from  learning  declarative  knowledge  for  its  own  sake,  declarative  knowledge  plays 

several,  potentially  important  roles  In  the  learning  of  procedural  knowledge.  Anderson  (1982), 
for  example,  suggests  that  the  learning  of  declarative  knowledge  Is  the  first  stage  involved  In 
the  acquisition  of  procedural  knowledge  and  that  we  learn  procedural  tasks  by  making  Inferences 
from  this  declarative  knowledge.  This  theory  has  a  great  deal  of  plausibility,  and  assuming  Its 

correctness,  research  Is  needed  on  the  type  of  declarative  knowledge  that  should  be  presented  In 

support  of  learning  the  types  of  procedural  knowledge  Involved  In  various  Air  Force  jobs  (or  job 
components  as  discussed  In  Gagne's  paper).  In  addition,  empirical  evidence  is  needed  that  is 
relevant  to  the  task  of  determining  how  we  can  best  facilitate  the  learning  of  both  the 

declarative  and  the  procedural  knowledge,  as  well  as  the  transition  between  them.  Anderson 

refers  to  this  stage  as  the  knowledge  compilation. 
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The  presentation  of  declarative  knowledge  Is  not  sufficient  in  and  of  Itself  for  the  desired 
learning  to  occur.  Learning  Is  an  active  process.  What  the  learner  does  Is  just  as  Important, 
If  not  more  important  than,  what  the  Instructor  does.  For  example,  cognitive  learning  theory 
emphasizes  the  crucial  role  played  by  the  particular  knowledge  or  schema  that  the  student  happens 
to  have  activated  at  the  time  learning  takes  place  (e.g.,  Bransford,  1979).  If  an  appropriate 
schema  Is  activated,  then  the  desired  learning  occurs  much  more  easily  than  would  be  the  case  If 
the  schema  had  not  been  activated.  Consequently,  various  types  of  Instructional  adjuncts,  aids 
to  comprehension  (e.g.,  Mayer,  In  press),  and/or  visual  mnemonics  (e.g..  Levin,  1981)  may  prove 
to  be  useful  Instructional  devices  for  Increasing  the  efficiency  of  learning  by  helping  to  elicit 
appropriate  schemata. 

As  far  as  retention  is  concerned,  there  appears  to  be  only  a  very  few  variables  that  affect 
memory  Independently  of  any  effect  that  the  variable  might  have  on  learning  (Shuell  4  Lee, 
1976).  Degree  of  original  learning,  of  course.  Is  one  of  these  variables,  and  anything  that 
Increases  how  well  something  Is  learned  will  consequently  have  an  Indirect  effect  on  how  well  It 
Is  remembered.  Another  variable  Is  distribution  of  practice  during  learning,  which  generally  has 
no  effect  (or  even  a  slight  decremental  effect)  on  learning  but  has  a  powerful  facllltatlve 
effect  on  memory  (Bloom  4  Shuell,  1981;  Shuell,  1981a).  The  third  variable  for  which  there  Is 
reasonably  good  evidence  Is  similarity  of  Interfering  material,  but  the  practical  Implications  of 
this  variable  are  not  clearcut. 

In  addition,  there  Is  some  suggestive,  but  at  present  rather  speculative,  evidence  that 
organization  (Shuell,  1981b)  and  the  use  of  multiple  contexts  during  learning  (Mackenzie  4  White, 
1982)  may  likewise  affect  memory  Independently  of  learning.  The  latter  variable  takes  on  added 
interest  when  one  considers  the  evidence  suggesting  that  transfer  Is  Increased  when  the  learner 
works  on  a  variety  of  tasks  during  learning  rather  than  working  on  a  single  task  (Duncan,  1958; 
Morrfsett  4  Hovland,  1959),  or  when  Irrelevant  Information  Is  present  during  learning  (Overlng  4 
Travers,  1966.  1967). 

These  various  considerations  lead  to  the  following  research  questions  (among  many  others) 
concerning  the  acquisition  and  retention  of  procedural  knowledge: 

1.  Do  any  of  the  following  aids  to  comprehension  facilitate  the  acquisition  of  the 
declarative  knowledge  learned  during  the  process  of  acquiring  procedural  knowledge  representative 
of  Air  Force  jobs,  thereby  enabling  the  desired  procedural  knowledge  to  be  learned  more  quickly? 

a.  Advance  organizers  of  the  concrete,  abstract  and  visual  types. 

b.  Visual  mnemonics. 

c.  Behavioral  objectives. 

d.  Adjunct  questions  of  both  the  written  and  oral  types. 

e.  Development  of  "cognitive  maps." 

2.  If  Instructional  adjuncts  such  as  those  listed  In  Question  No.  1  are  developed  for  the 
procedural  knowledge  Itself  (rather  than  the  declarative  knowledge  supporting  It)  and  used  during 
the  "Knowledge  Compilation"  and/or  "Procedural"  stages,  will  the  acquisition  and/or  retention  of 
the  desired  procedure  be  facilitated? 

3.  Ooes  the  learning  of  procedural  knowledge  under  conditions  of  distributed  practice 
facilitate  the  long-term  retention  of  the  procedure? 
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Several  aspects  of  Individual  differences  In  learning  from  Instruction  have  already  been 
discussed  In  this  paper  and  will  not  be  repeated  here.  What  will  be  presented  In  this  section 
are  some  possible  research  questions  based  on  the  concerns  and  research  discussed  earlier.  In  a 
training  environment  such  as  the  present  one,  a  concern  for  Individual  differences  Implies  at 
some  level  a  concern  for  Individualization  of  Instruction.  In  some  cases,  however,  these 
Instructional  concerns  may  be  primarily  Indirect  or  postponed  to  future  research  and 
development.  Although  the  ATI  paradigm  (and  the  extensions  of  It  discussed  In  an  earlier  section 
of  this  paper)  Is  the  most  explicit  and  Integrated  approach  In  this  regard,  research  questions 
over  and  above  those  suggested  earlier  will  not  be  developed  here  due  to  space  limitations.  Such 
development  would  Involve  a  fairly  extensive  statement  In  order  to  do  an  adequate  job,  and  It  Is 
not  feasible  to  go  Into  the  necessary  detail  In  this  section.  Consequently,  the  following 
research  questions  represent  a  somewhat  selective  part  of  the  programmatic  research  on  Individual 
differences  that  should  be  conducted  at  the  laboratory: 

1.  In  what  ways  do  Individuals  who  are  experts  In  performing  various  Air  Force  jobs  differ 
from  new  trainees  with  regard  to  the  following: 

a.  The  way  the  task  demands  of  the  job  are  perceived? 

b.  The  way  the  Individual  goes  about  performing  the  job  to  the  best  of  his  or  her  ability? 

c.  Variables  that  differentially  affect  the  performance  of  "experts,"  "apprentices,"  and 
"novices"  (In  order  to  determine  Instructional  Interventions  that  will  most  efficiently  help  new 
and  advanced  trainees  to  become  more  proficient  at  performing  their  jobs)? 

2.  What  variables  differentially  affect  the  performance  of  "good"  and  "poor"  learners  at 
various  levels  of  competence,  such  as  "novices"  and  "apprentices"  (so  that  training  procedures 
can  be  developed  that  will  enable  all  personnel  to  learn  and  retain  assigned  tasks  to  the  maximum 
level  possible)? 


Conclusion 

Several  themes  of  current  research  on  cognitive  psychology  and  learning  from  instruction  have 
been  discussed  within  the  context  of  Air  Force  training.  These  themes  will  undoubtedly  influence 
the  research  conducted  by  the  proposed  learning  research  laboratory  and  should  be  considered  in 
planning  the  research  agenda  for  the  laboratory.  This  paper  has  tried  to  identify  some  of  the 
topics  that  might  be  incorporated  into  various  research  projects.  For  example,  further 
Investigation  is  needed  on  the  instructional  factors  that  help  to  maintain  and  support  the 
retention  of  skills  and  transfer;  but  we  need  to  relate  these  factors  to  the  various  objectives 
of  Air  Force  training  and  try  to  analyze  on-the-job  types  of  experiences,  not  only  in  terms  of 
logical  categories,  but  also  In  terms  of  psychological  categories  that  have  relevance  for 
learning  from  Instruction. 
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CHAPTER  6 


APPROACHES  TO  THE  ACQUISITIOH  OF  UNDERSTANDING 
Janes  N.  Royer 

University  of  Massachusetts,  Amherst 


This  paper  outlines  a  strategy  for  the  development  of  Instructional  procedures  that  will 
assure  students  have  understood  the  Instruction  they  have  received.  The  paper  begins  with  a 
description  of  a  taxonomy  of  educational  problems,  followed  by  a  discussion  of  what  It  means  to 
understand  from  the  perspective  of  cognitive  theory.  This  Is  followed  by  the  suggestion  that 
satisfactory  performance  on  transfer  objectives  will  provide  an  Index  of  the  extent  to  which 
students  have  understood  the  Instruction  they  have  received.  Given  the  acceptance  of  the 
transfer  objective  as  an  operational  definition  of  understanding,  the  paper  then  goes  on  to 
describe  four  techniques  that  facilitate  the  understanding  of  Instruction.  This  section  Is 
followed  by  a  description  of  research  relevant  to  keeping  students  Involved  In  learning  tasks  and 
by  a  section  describing  the  Importance  of  assessing  knowledge  In  specific  content  areas.  The 
next  section  of  the  paper  presents  a  revision  of  the  ISO  model  that  Incorporates  many  of  the 
suggestions  contained  In  this  paper.  The  final  section  of  the  paper  contains  suggestions  for 
specific  research  projects  that  the  Air  Force  could  undertake  In  a  learning  research  laboratory. 


A  Taxonomy  of  Educational  Problems 


By  nature,  I  am  the  sort  of  person  who  attempts  to  Impose  some  boundaries  and  structure  on  a 
problem  before  I  can  think  It  through.  Hence,  I  began  my  task  on  this  project  by  setting  up  a 
kind  of  classification  system  that  Is  similar  to  one  I  and  others  have  used  previously  (e.g., 
Rothkopf,  1968;  Royer  A  Allan,  1978).  The  gist  of  this  system  Is  that  educational  problems  can 
be  divided  Into  three  types:  problems  Involving  observable  student  behaviors,  problems  Involving 
the  mastery  of  basic,  low-level  Information,  and  problems  Involving  the  understanding  of  more 
complex  materials. 

Problems  Involving  observable  behaviors  most  often  entail  getting  students  to  engage  In 
task-appropriate  behavior  and  getting  them  not  to  engage  In  task-inappropriate  behavior.  Most 
problems  of  this  sort  can,  I  believe,  be  successfully  dealt  with  through  the  application  of 
behavior  modification  principles  and/or  through  the  use  of  task  analysis  procedures. 

The  second  type  of  problem  entails  the  mastery  of  basic  low-level  performance.  I  have  In 
mind  here  the  mastery  of  low-level  terms,  definitions,  quantitative  entitles,  and  concepts  that 
provide  the  foundation  of  any  area  of  study.  Here  again,  I  think  we  have  powerful  psychological 
machinery  that  can  be  activated  to  assist  In  acquiring  Information  of  this  sort.  I  think 
associative  learning  theory  can  tell  us  a  great  deal  about  setting  up  Ideal  conditions  for  the 
mastery  of  basic  Information.  Associative  learning  theory  is  out  of  vogue  now,  but  we  should  not 
forget  that  It  tells  us  a  good  deal  about  practice  schedules,  review  Intervals,  and  conditions  of 
transfer,  all  of  which  are  valuable  components  of  an  Instructional  sequence  designed  to  assure 
mastery  of  basic  Information.  If  we  embed  the  principles  of  associative  learning  theory  within  a 
design  procedure  for  the  systematic  production  of  Instructional  materials,  such  as  the  familiar 
one  In  Figure  6-1,  we  have  a  particularly  powerful  method  of  developing  effective  Instruction. 
The  principles  of  associative  learning  theory  provide  the  guideline.;  fcr  instructional 
development,  and  the  systems  model  provides  the  framework  under  which  this  d»v«'lo3«ent  occurs. 
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We  now  turn  to  the  third  of  qy  categories  -  problems  Involving  understanding.  Understanding 
is  one  of  those  terms  that  the  early  proponents  of  behavioral  objectives  urged  us  to  purge  from 
our  vocabularies  because  It  was  so  hopelessly  vague  that  It  could  never  serve  as  a  meaningful 
educational  goal  (Hager .  1962);  but  our  understanding  of  what  It  means  to  understand  has 
Increased  considerably  In  recent  years,  principally  because  the  question  of  what  It  means  to 
understand  1$  at  the  very  heart  of  rapidly  developing  cognitive  theory. 


Understanding  from  a  Cognitive  Perspective 

Hy  favorite  way  of  Illustrating  the  change  In  the  way  we  think  about  the  process  of 
understanding  Is  to  Invoke  the  contrasting  metaphors  the  linguist  Michael  Reddy  has  used  to 
Illustrate  the  erroneous  Implicit  belief  we  have  about  meaning  In  language.  Reddy  (1979)  has 
pointed  out  that  the  way  we  describe  meaning  documents  the  existence  of  an  unconscious  acceptance 
of  what  he  has  called  the  conduit  metaphor  of  meaning  In  language.  Consider  the  following 
examples  which  are  merely  a  smattering  of  the  hundreds  of  examples  Reddy  has  collected.  "Did  you 
catch  what  I  meant?"  "Your  Ideas  just  don't  come  across  In  your  writing."  “Your  speech 
effectively  conveyed  your  beliefs."  "Did  you  get  the  author's  message?" 

Reddy  has  argued  that  the  examples  above  reflect  the  acceptance  of  a  model  of  meaning  that 
says  that  the  sender  of  a  message  packages  the  meaning  of  the  message  In  the  form  of  speech  or 
writing,  sends  that  message  through  some  communication  conduit,  whereupon  the  receiver  of  the 
message  unpackages  It,  and  extracts  the  meaning.  Reddy  argues  that  acceptance  of  the  conduit 
metaphor  means  one  accepts  the  position  that  the  meaning  of  a  message  resides  within  the  message 
Itself.  Incldently,  the  belief  that  meaning  resides  within  a  message  Is  an  Integral  part  of 
associative  learning  theory.  Within  the  framework  of  associative  learning  theory,  memory  Is 
merely  the  blackboard  that  experience  writes  on. 

Reddy  argued  that  the  conduit  metaphor  was  wrong,  and  he  offered  a  substitute  that  he  called 
"the  tool -maker  metaphor,"  which  he  believed  to  more  accurately  represent  the  process  of  deriving 
meaning.  Reddy  began  his  metaphor  with  the  suggestion  that  communication  between  Individuals  was 
analogous  to  Interactions  between  people  In  slightly  different  environments.  This  situation 
could  be  imagined  by  thinking  of  people  who  live  in  separate  compartments  of  a  huge  compound  that 
Is  shaped  like  a  wagon  wheel.  He  suggested  that  we  Imagine  each  wedge-shaped  sector  of  the  wheel 
is  an  environment  that  has  much  In  common  with  one  another  -  water,  trees,  plants,  rocks,  and  the 
like  -  but  none  of  the  environments  Is  exactly  alike.  The  trees  are  different;  one  environment 
Is  swampy  and  another  Is  dry  and  rocky,  and  so  on.  At  the  hub  of  the  wheel  there  Is  a  device 
that  delivers  sheets  of  paper  to  people  in  the  other  environments.  This  Is  the  only  means  of 
communication;  the  people  cannot  communicate  directly,  and  they  cannot  experience  each  other's 
environment.  Reddy  then  suggests  that  one  person,  say  the  person  living  In  environment  A, 
discovers  how  to  build  a  wooden  rake  to  clear  dead  leaves  and  other  debris  away  from  his 
cultivated  plants  and  that  he  decides  to  share  his  discovery  with  the  residents  of  the  other 
environments.  He  draws  the  plans  for  his  rake  on  a  sheet  of  paper  and  drops  the  plans  through 
the  slots  for  delivery  to  the  people  In  environments  B,  C,  and  D.  The  people  living  In  the  other 
environments  receive  the  sheets  and  begin  to  construct  what  they  can  from  them. 

Environment  A  contains  a  great  many  trees,  which  Is  why  the  rake  was  made  of  wood  and  was 
useful  for  gathering  fallen  leaves.  The  environment  In  sector  B,  however.  Is  mostly  rock  and 
contains  few  trees.  The  person  living  in  environment  B  uses  rock  In  all  of  his  construction,  and 
after  not  being  able  to  find  a  piece  of  wood  suitable  for  a  handle,  he  begins  to  construct  a  rake 
head  out  of  stone.  The  resulting  implement  Is  too  heavy  to  be  lifted;  moreover,  he  cannot 
Imagine  any  possible  use  for  It.  Suddenly,  he  gets  an  Idea.  If  the  stone  head  is  modified  so 
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that  it  consists  of  only  two  large  prongs.  It  will  be  useful  In  digging  stones  from  the  field. 
Person  B  Makes  his  new  Implement  and  Is  very  happy  with  the  result. 

Environment  C  Is  wet  and  swanpy,  and  the  person  living  there  cannot  see  any  use  for  the 

device  Invented  by  person  A.  However,  after  some  experimenting,  he  devises  a  hoe  which  Is  very 

useful  for  cutting  the  rampant  plant  growth  that  threatens  his  garden. 

Reddy  continues  on  with  his  story,  but  at  this  point  we  can  see  the  direction  he  Is  taking. 

The  Individuals  exist  In  environments  that  are  similar  enough  that  they  recognize  the  general 

purpose  of  the  rake  plans,  and  each  Individual  has  received  exactly  the  same  message.  However, 
the  Interpretation  of  the  message  differs  because  of  the  differing  environments.  A  rake  becomes 
a  stone-picker  or  a  hoe  because  these  are  the  only  tools  that  make  sense  In  terms  of  environments 
B  and  C. 

Reddy  suggests  that  the  nature  of  the  language  communication  process  Is  more  similar  to  the 
tool -maker  metaphor  than  It  Is  to  the  conduit  metaphor.  We  do  not  "receive"  the  meaning  of  a 
message  passed  along  some  conduit.  Instead,  we  must  construct  a  meaning  by  Interpreting  a 
message  In  light  of  our  knowledge  (cf.  Bransford  A  McCarrell ,  1974).  Thus,  the  process  of 
understanding  Involves  an  Interaction  between  a  received  linguistic  message,  the  context  In  which 
a  message  1$  received,  and  the  knowledge  background  of  the  person  receiving  the  message  (Jenkins, 
1979). 

The  constructivist  Interpretation  Is  at  the  core  of  the  cognitive  explanations  of 
understanding.  Something  Is  understood  when  It  has  been  Integrated  In  a  meaningful  way  Into  the 
learner's  existing  knowledge  structure.  When  the  learner  does  not  have  any  relevant  knowledge 
that  he  can  use  to  construct  an  Interpretation,  memorization  may  occur,  but  understanding  will 
not. 

The  argument  that  understanding  involves  the  Integration  of  new  material  Into  an  existing 
knowledge  structure  Is  an  Interesting  theoretical  statement;  and,  as  I  will  soon  argue,  it 

provides  guidelines  for  developing  approaches  to  enhance  understanding,  but  It  leaves  open  the 

question  of  how  we  will  know  when  a  student  has  understood  what  we  are  trying  to  teach.  I  would 

like  to  suggest  a  slightly  different  type  of  Instructional  objective  as  the  answer  to  this 

question. 


Transfer  Objectives  as  an  Index  of  Understanding 

One  powerful  element  of  the  technology  of  instruction  approach  popular  during  the  1960s  was 
the  use  of  behavioral  objectives  as  a  standard  for  instructional  success.  An  Instructional 
approach  was  deemed  successful  when  students  receiving  the  Instruction  performed  In  the  manner 
specified  In  the  Instructional  objective.  I  think  we  should  adopt  a  similar  standard  of 
performance  to  evaluate  the  success  of  Instructional  programs  based  on  cognitive  research  and 
theory.  The  nature  of  the  objectives  will,  however,  be  different.  A  common  standard  for 
evaluating  whether  a  student  has  understood  something  is  whether  the  student  can  use  what  has 
been  learned.  By  use,  I  mean  the  demonstration  that  the  student  can  perform  some  meaningful 
activity  beyond  the  replication  of  the  instruction  as  experienced.  Thus,  repeating  or 
recognizing  an  aspect  of  Instruction  would  not  constitute  evidence  of  understanding,  whereas 
using  the  Instruction  as  a  tool  for  the  acquisition  of  further  Instruction,  or  successfully  using 
the  Instruction  In  an  unfamiliar  context,  would  be  evidence  that  the  Instruction  has  been 
understood. 


Thus,  I  m  suggesting  that  objectives  which  Index  the  transfer  of  Instruction  serve  as  the 
standard  for  judging  the  success  of  Instructional  approaches  designed  to  enhance  understanding. 
Transfer  objectives  can  range  from  specifications  that  differ  only  slightly  from  the  conditions 
of  original  Instruction  to  those  that  have  only  conceptual  relations  to  the  Instructions  the 
students  received.  An  example  of  the  former  kind  of  transfer  task  Is  Anderson’s  (1972) 
recommendation  that  comprehension  of  Instruction  be  assessed  by  determining  If  students  can 
detect  paraphrases  of  content  experienced  during  Instruction.  An  exanple  of  a  more  distant 
transfer  objective  would  be  to  design  a  specific  electronic  component  after  completing  a  course 
In  basic  electronics. 


proaches  to  Enhancing  Understands 


Given  that  the  goal  of  Instructional  efforts  based  on  cognitive  theory  Is  to  develop 
understanding,  and  given  that  understanding  can  be  assessed  by  determining  If  Instruction 
transfers,  the  next  question  Is,  "What  does  cognitive  research  and  theory  provide  In  the  way  of 
guidelines  for  developing  Instruction  that  will  encourage  understanding?"  First,  there  Is  a  very 
general  answer  to  this  question:  Understanding  will  be  enhanced  when  the  Instructional  material 
can  be  Integrated  Into  the  learner's  existing  knowledge  structure.  However,  this  general  answer 
breaks  down  Into  three  more  specific  answers.  The  first  of  these  Is  that  understanding  can  be 
enhanced  by  Increasing  the  likelihood  that  students  will  bring  appropriate  prior  knowledge  to 
bear  when  receiving  Instruction.  Second,  understanding  can  be  enhanced  by  providing  the  student 
with  new  Information  that  will  assist  In  Integrating  the  Instructional  material  Into  that 
student's  existing  knowledge  structure.  Third,  understanding  can  be  enhanced  by  teaching 
students  beneficial  learning  strategies  and  by  assuring  that  the  students  use  the  strategies.  I 
will  now  discuss  each  of  these  specific  answers  In  turn. 


Activating  Existing  Prior  Knowledge 


The  first  approach  to  encouraging  understanding  Is  to  activate  knowledge  that  the  students 
already  have,  so  that  Incoming  Information  can  then  be  Integrated  Into  the  activated  structure. 
One  of  the  simplest  techniques  for  doing  this  Is  the  use  of  mnemonics.  The  most  thoroughly 
documented  educational  use  of  mnemonic  procedures  Is  a  technique  called  the  keyword  method. 


The  keyword  method  as  an  example  of  a  mnemonic  technique.  The  term  "keyword  method"  was 
coined  by  Richard  Atkinson  (Atkinson,  1975)  to  describe  the  technique  of  foreign  language 
vocabulary  learning  that  Involves  associating  the  foreign  word  with  a  well-known,  concrete 
English  word,  which  Is  called  the  keyword.  The  keyword  need  not  have  a  meaning  relationship  with 
the  foreign  word.  It  must,  however,  be  similar  In  sound.  For  Instance,  the  keyword  used  by 
Raugh  and  Atkinson  (1975)  to  teach  the  Spanish  word  for  duck  (pato  -  pronounced  pot-o)  was 
"pot."  As  another  example,  the  keyword  for  learning  the  Spanish  word  for  horse  (cabal lo  - 
pronounced  cob-eye-yo)  was  "eye." 


The  keyword  method  divides  vocabulary  learning  Into  two  stages.  The  first  step  requires  the 
student  to  associate  the  spoken  foreign  word  with  the  keyword.  This  association  Is  generally 
learned  quickly  because  of  the  sound  similarity  between  the  foreign  word  and  the  keyword.  The 
second  stage  requires  the  subject  to  form  a  mental  Image  of  the  keyword  "Interacting"  with  the 
English  translation  of  the  foreign  word.  For  example,  the  student  might  form  an  Image  of  a  duck 
with  a  pot  on  Its  head  as  an  aid  In  learning  the  meaning  of  "pato”  or,  another  example,  to 
Imagine  a  horse  with  a  large  eye  In  the  middle  of  Its  forehead  as  an  aid  In  learning  the  meaning 
of  "cabal lo." 


The  keyword  technique  has  proven  to  be  a  remarkably  effective  technique  for  acquiring  foreign 
language  vocabulary  words  In  a  variety  of  settings  and  with  a  number  of  different  kinds  of 
students.  Atkinson  and  Raugh  (1975)  had  one  group  of  Stanford  University  undergraduates  study  a 
120-word  Russian  vocabulary  list  using  the  keyword  method  while  a  control  group.  Instructed  to 
learn  the  words  In  any  way  they  chose,  studied  the  same  list.  At  the  end  of  a  set  period  of 
time,  the  group  using  the  keyword  technique  had  learned  an  average  of  55  of  the  words.  In 
addition,  this  learning  advantage  persisted  over  time.  When  tested  6  weeks  later,  the  keyword 
group  remembered  the  meaning  of  43S  of  the  120  words,  whereas  the  control  group  remembered  only 
281  of  the  words.  In  another  study  Involving  college  students,  Raugh  and  Atkinson  (1975) 
reported  that  a  group  using  the  keyword  method  mastered  881  of  a  60-word  Spanish  vocabulary  list, 
whereas  a  control  group  studying  the  list  for  the  same  period  of  time  mastered  only  281  of  the 
list. 

The  keyword  method  has  also  proven  to  be  a  very  effective  means  of  learning  foreign  language 
vocabulary  with  much  younger  students.  Pressley  and  Levin  (1978),  and  Levin,  Pressley, 
McCormick,  Miller,  and  Shrlberg  (1979),  have  reported  a  number  of  experiments  that  Involve 
teaching  children  In  elementary  school  the  meaning  of  Spanish  words.  In  every  study  Involving 
children  In  this  age  group,  those  students  who  learned  the  words  using  the  keyword  method  learned 
the  meaning  of  more  words  than  did  students  who  used  their  own  methods  to  learn  the  words. 
Moreover,  the  margin  of  advantage  for  the  keyword  students  was  frequently  greater  than  2:1. 

Thus  far,  the  reviewed  research  has  focused  on  the  acquisition  of  foreign  words.  We  will  now 
consider  evidence  that  not  only  does  the  keyword  method  produce  superior  acquisition.  It  also 
meets  the  criterion  for  understood  learning  In  that  It  produces  superior  transfer.  Pressley, 
Levin,  and  Miller  (1980)  reported  several  studies  In  which  college  students  learned  the 
definitions  of  Infrequently  used  English  words  using  either  the  keyword  technique  or  a  technique 
of  the  student's  own  devising.  The  students  then  performed  a  variety  of  tasks  that  Included 
Identifying  when  a  targeted  word  was  used  correctly  or  Incorrectly  In  a  sentence,  selecting  the 
correct  sentence  Into  which  a  targeted  word  could  be  Inserted,  and  constructing  a  sentence  that 
Involved  the  correct  usage  of  a  targeted  word.  In  each  of  these  transfer  tasks,  the  group  that 
learned  the  words  using  the  keyword  technique  performed  at  a  level  superior  to  that  of  the 
control  group. 

Using  mnemonics  represents  one  of  the  simplest  means  of  activating  prior  knowledge  as  an  aid 
In  achieving  meaningful  learning  of  new  material.  The  research  reviewed  focuses  on  the 
acquisition  of  vocabulary  words,  but  It  should  be  noted  that  the  technique  could  be  used  for 
acquiring  a  variety  of  kinds  of  Information  such  as  technical  names  for  objects  or  other  things, 
and  the  names  of  people  or  places  (Lorayne  t  Lucas,  1974). 

Using  analogies  to  facilitate  subsequent  learning.  Another  way  of  activating  prior  knowledge 
that  can  then  be  used  to  facilitate  subsequent  learning  Is  through  the  use  of  analogies.  Royer 
and  his  associates  (Royer  A  Cable,  1975,  1976)  have  reported  several  studies  using  this 
technique.  They  wrote  two  passages,  one  of  which  was  concerned  with  the  transfer  of  electricity 
through  metals;  the  other  was  concerned  with  the  transfer  of  heat  through  metals.  Each  of  these 
passages  was  then  rewritten  in  two  forms.  One  form  -  called  the  "concrete"  form  -  was  written  In 
an  easily  understandable  form  that  made  extensive  use  of  analogies  to  assist  in  understanding 
difficult  concepts.  The  other  version  -  called  the  "abstract"  version  -  was  purposely  written  in 
a  highly  abstract  style  that  was  supposed  to  be  difficult  to  understand. 

Royer  and  Cable  (1975)  chose  heat  and  electrical  conduction  through  metals  as  topics  for 
their  passages  because  of  the  considerable  similarities  between  the  two  phenomena.  Metals  are 
excellent  conductors  of  both  heat  and  electricity  because  they  are  crystalline  In  structure,  and 


they  have  a  great  aany  free-floating  electrons.  These  properties  were  explained  by  analogy  In 
the  following  way: 

Heat  transfer  actually  Involves  the  transfer  of  molecular  motion.  In  the  case  of 
heat  conduction  through  metals,  this  transfer  of  motion  occurs  through  a  solid 
substance  such  as  a  bar  of  Iron.  If  we  were  able  to  examine  a  bar  of  Iron  through 
an  extremely  powerful  microscope,  we  would  see  that  the  Interior  consists  of  a 
series  of  regularly  shaped  and  spaced  structural  units  known  as  crystal  lattices. 

In  order  to  picture  these  lattices,  Imagine  a  box  made  of  many  Tinker  Toys  with 
smaller  boxes  Inside  consisting  of  other  joined  Tinker  Toys.  The  solid  round  parts 
of  the  Tinker  Toys  would  correspond  to  the  molecules  within  the  crystal  lattice.  In 
our  bar  of  Iron,  which  Is  a  good  conductor  of  heat,  each  of  the  bonded  molecules 
within  the  lattice  has  associated  with  It  several  "free -floating"  electrons.  Each 
crystal  lattice  then  Is  an  orderly  array  of  molecules  surrounded  by  a  cloud  of 
electrons  that  are  not  attached  to  any  particular  molecule,  but  are  free  to  move  at 
random  through  the  lattice.  You  can  picture  this  by  Imagining  many  tiny  particles 
floating  through  the  series  of  Tinker  Toy  boxes. 

By  using  analogies  such  as  the  Tinker  Toy  analogy  In  the  passage  above,  Royer  and  Cable 
(1975)  hoped  to  activate  prior  knowledge  that  could  be  used  to  facilitate  the  understanding  of 
the  material  being  presented  In  the  passages.  This  understanding  could  then  be  assessed  by 
presenting  the  students  with  related  material  and  seeing  If  what  they  had  learned  previously 
eased  the  learning  of  the  related  material;  In  short,  determining  whether  the  learned  material 
would  positively  transfer  to  the  learning  of  additional  material. 

The  results  of  an  experiment  supported  the  prediction  that  activating  knowledge  by  using  an 
analogy  would  ease  the  learning  of  subsequent  related  material.  Students  who  read  the  analogy 
material  prior  to  reading  a  second  passage  learned  at  least  401  more  from  the  second  passage  than 
did  a  group  that  did  not  read  the  analogy  material. 

The  results  of  the  Initial  Royer  and  Cable  (1975)  study  have  been  replicated  many  times 
(Perkins,  1978;  Royer  4  Cable,  1976;  Royer  6  Perkins,  1977).  In  addition,  Royer  and  Perkins 
(1977)  were  able  to  demonstrate  that  the  meaningful  learning  produced  by  using  analogies 
persisted  over  time.  They  presented  college  students  with  an  Initial  passage  with  analogies,  or 
with  a  control  passage;  and  they  then  had  the  students  read  a  second  transfer  passage 
Immediately,  after  2  days,  or  after  1  week.  Their  results  Indicated  that  the  positive  transfer 
attributable  to  using  analogies  was  as  strong  after  1  week  as  It  was  Immediately. 

Coming  up  with  exactly  the  right  analogy  to  achieve  meaningful  learning  of  conceptually 
difficult  material  Is  a  truly  creative  act.  This  Is  especially  true  since  many  analogies  do  not 
work  very  well.  Describing  a  living  organism,  for  example,  as  being  like  a  factory  that  takes  In 
raw  materials  and  puts  out  waste  products  may  capture  one  attribute  of  living  organisms;  but  the 
"fit*  of  the  analogy  Is  so  poor  that  It  may  generate  so  much  confusion  that  It  negates  any 
benefits  that  might  occur  with  Its  use.  For  example,  a  discerning  student  might  point  out  that 
one  of  the  defining  features  of  factories  Is  that  they  produce  a  product.  If  living  organisms 
are  like  factories,  what  "products"  do  they  produce? 

Another  caution  with  using  analogies  to  assist  in  achieving  meaningful  learning  Is  that  the 
supposedly  known  part  of  the  analogy  must.  In  fact,  be  part  of  the  learner's  knowledge.  If  we 
told  you  that  Aplysla  (a  large  sea  snail)  Is  a  favorite  organism  of  study  by  neurobiologists 
because  It  has  a  characteristic  of  Its  nervous  system  that  Is  similar  to  the  also  favored  squid, 
chances  are  that  we  would  not  have  enlightened  you  In  the  least.  Only  If  you  know  that  squid 
have  very  large  nerve  cells,  thereby  making  them  easy  to  examine  and  study,  would  you  benefit 
from  being  told  that  Aplysla  Is  a  favorite  organism  of  study  for  the  same  reason.  Teachers 
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should  be  very  careful  to  sake  sure  that  the  supposedly  known  part  of  the  analogy  Is,  In  fact, 
part  of  the  learner's  knowledge.  Otherwise,  using  the  analogy  certainly  will  not  help  the 
learner,  and.  In  fact,  say  be  worse  than  nothing  In  that  learners  now  have  two  things  they  do  not 
understand  rather  than  one. 

Questions  such  as  the  ones  mentioned  above  are  very  fruitful  Issues  for  future  researchers. 
Moreover,  analogy  research  Is  receiving  Increasing  attention  In  both  the  basic  and  applied 
literatures.  An  exaaple  of  the  fomer  Is  the  Insightful  analysis  recently  presented  by  Glck  and 
Holyoak  (1983).  On  the  more  applied  side,  a  paper  written  by  a  group  of  Dutch  psychologists 
examines  the  educational  uses  of  analogies  and  will  soon  appear  in  the  Journal  of  Educational 
Psychology. 


Enhancing  Meaningful  Learning  by  Teaching  Students  Information 
that  Hill  Ease  Subsequent  Learning 


In  the  previous  section,  I  discussed  techniques  for  achieving  meaningful  learning  by 
activating  Information  the  student  has  already  learned.  In  this  section,  I  examine  techniques 
for  achieving  meaningful  learning  that  Involves  teaching  the  student  new  Information  that  can 
then  be  used  to  ease  subsequent  learning. 

Advance  organizers.  In  the  early  1960s,  David  Ausubel  (1960,  1963)  suggested  advance 

organizers  as  a  device  for  easing  the  learning  of  subsequently  presented  material.  Advance 

organizers  are  statements  that  are  "presented  at  a  higher  level  of  abstraction,  generality,  and 
Inclusiveness  than  the  new  material  to  be  learned"  (Ausubel,  Novak,  A  Haneslon,  1978,  p.  171). 
Ausubel  envisioned  advance  organizers  as  a  bridge  between  "what  the  learner  already  knows  and 
what  he  needs  to  know  before  he  can  meaningfully  learn  the  task  at  hand"  (Ausubel,  Novak,  A 
Haneslon,  1978,  p.  172). 

Ausubel  and  his  fellow  researchers  conducted  many  studies  during  the  1960s  (Ausubel,  1960; 
Ausubel  A  Fitzgerald,  1961,  1962;  Ausubel  A  Youssef,  1963;  Fitzgerald  A  Ausubel,  1963)  that 
demonstrated  the  efficacy  of  presenting  students  with  advance  organizers  prior  to  engaging  In  a 
learning  task.  A  typical  study  by  Ausubel  and  Youssef  (1963)  Involved  having  college  students 
read  a  passage  on  Buddhism  after  reading  a  "comparative  advance  organizer"  (examining  the 
relation  between  Buddhism  and  Christianity)  or  a  control  passage.  They  found  that  the  students 
who  had  initially  read  the  organizer  passage  retained  more  from  the  Buddhism  passage  than  did  the 
students  who  Initially  read  the  control  passage.  In  addition  to  Ausubel's  studies  which  have 

shown  greater  acquisition  of  subsequently  presented  material,  other  studies  have  shown  beneficial 

effects  of  advance  organizers  using  more  remote  transfer  tasks.  Grotelueschen  and  Sjogren 
(1968),  Merrill  and  Stolurow  (1966),  and  Scandura  and  Wells  (1967)  have  all  shown  that  material 
preceded  by  advance  organizers  Is  more  likely  to  transfer  to  remote  tasks  than  material  not 
preceded  by  advance  organizers. 

At  this  point,  there  seems  to  be  little  doubt  that  advance  organizers  can  improve  the 
learning  of  subsequent  material  under  certain  conditions.  Ausubel  (Ausubel,  Novak,  A  Hanesion, 
1978)  has  Indicated  two  of  these  conditions.  First,  the  material  to  be  presented  must  involve 
unitary  topics  or  related  sets  of  Ideas.  If  the  ideas  to  be  presented  are  highly  varied,  as 
might  be  the  case,  for  example,  in  a  college  level  survey  class  (e.g.,  introductory  psychology), 
then  it  would  not  be  possible  to  generate  a  single  advance  organizer  to  encompass  all  of  the 
ideas.  This  does  not  preclude,  however,  breaking  the  subject  matter  into  unitary  topics  and  then 
preparing  advance  organizers  for  each  topic. 
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Ausubel's  second  circumstance  Is  that  a  true  advance  organizer  Bust  be  developed  given 
knowledge  of  the  learner's  existing  knowledge  structure.  This  means  that  If  advance  organizers 
are  to  serve  as  a  bridge  between  material  to  be  presented  and  material  the  learner  already  knows. 
It  Is  essential.  In  fact,  to  know  what  the  learner  already  knows.  Otherwise,  a  teacher  might  end 
up  presenting  a  learner  with  two  things  (the  advance  organizer  and  the  target  material)  that  the 
student  has  difficulty  In  learning  rather  than  just  one  (the  target  material). 

Models  as  a  fora  of  concrete  advance  organizer.  Ausubel  aeflned  an  advance  organizer  as  a 
statement  that  was  at  a  higher  level  of  abstraction  than  the  target  material  to  follow.  Another 
kind  of  advance  organizer  that  has  received  considerable  Investigation  is  the  "model"  or 
"concrete"  advance  organizer. 

A  model  or  concrete  advance  organizer  Is  either  a  statement  or  an  Illustration  that  provides 
a  concrete  general  overview  of  a  system  the  student  is  going  to  study  in  detail.  An  example  of 
the  use  of  a  model  as  an  advance  organizer  can  be  found  In  Mayer's  research  on  teaching  college 
students  a  computer  programming  language  (Mayer,  1975,  1976;  Mayer  A  Bromage,  1980).  Mayer 
(1975)  presented  his  "model"  students  with  a  text  that  began  with  a  diagram  of  the  Inside  working 
components  of  a  computer.  The  diagram  Included  an  "Input  window"  (described  as  a  ticket  window) 
and  "output  pad"  (described  as  a  pad  of  message  paper),  a  "memory  scoreboard"  (described  as  an 
e1ght>space,  erasable  scoreboard),  and  a  "program  list  with  pointer  arrow"  (described  as  a 
shopping  list).  The  use  of  descriptions  such  as  a  "ticket  window"  for  the  Input  window  was 
designed  to  make  the  unfamiliar  components  more  familiar  by  relating  them  to  Information  the 
students  already  possessed.  After  the  students  had  become  familiar  with  the  model,  they  received 
instruction  on  computer  programming  that  made  continued  reference  to  the  model.  In  effect,  the 
student  was  asked  to  "role  play"  what  the  computer  did  with  each  of  the  programing  statements. 

Another  group  of  students  In  Mayer's  (1975)  study  were  Instructed  on  computer  programming  In 
a  more  traditional  manner.  They  were  given  definitions  of  programing  statements  and  examples  of 
the  appropriate  use  of  each  of  the  statements,  following  Instruction,  each  of  the  groups  was 
asked  to  write  a  number  of  short  computer  programs  that  varied  in  type  of  problem  and  In 
complexity.  A  comparison  of  the  performance  of  the  groups  revealed  that  students  who  had 
received  the  model  as  an  advance  organizer  performed  significantly  better  on  the  programming 
transfer  task  than  did  students  who  received  the  traditional  Instruction.  The  results  were 
subsequently  replicated  In  later  studies  (Mayer,  1976;  Mayer  A  Bromage,  1980), 

Mayer's  research  on  computer  prograiranl ng  represents  one  Instance  where  a  system  or  process 
model  can  serve  as  a  highly  beneficial  advance  organizer.  In  fact,  models  can  be  beneficial  In 

any  area  of  study  Involving  a  concrete  system.  Imagine,  for  example,  that  you  were  teaching  an 

automotive  repair  course.  A  good  strategy  would  be  to  begin  the  course  with  a  model  overview  of 
the  functioning  of  an  automobile.  Moreover,  It  would  probably  be  beneficial  to  review  the  model 
as  the  students  move  from  one  system  to  another.  So,  for  example.  If  the  students  had  been 
working  on  the  carburetor  and  were  ready  to  move  to  the  ignition  system,  it  would  probably  be 
worthwhile  to  present  the  model  of  the  entire  car  again  so  that  they  could  relate  the  functioning 
of  the  Ignition  system  to  the  operation  of  the  entire  car. 

Techniques  for  Maintaining  Active  Cognitive  Processing 

In  the  sections  above,  I  discussed  what  it  meant  to  understand,  and  techniques  for  enhancing 
understanding.  However,  there  Is  another  problem  In  the  general  area  of  Instructing  for 

understanding.  This  Is  the  problem  of  how  to  maintain  an  appropriate  level  of  cognitive 

processing  and  how  to  focus  attention  on  the  Instructional  content  that  is  believed  to  be  most 
important  (cf.  Royer,  Bates,  A  Konold,  1983). 


Giving  students  Instructional  objectives.  When  Instructors  teach  a  course,  they  do  not 
expect  the  student  to  ouster  all  of  the  Information  presented.  Instead,  they  generally  have 
specific  goals  that  they  would  like  the  students  to  attain,  and  only  part  of  the  Information 
presented  will  be  directly  relevant  to  those  goals.  The  material  not  directly  relevant  to  the 
goals  Is  presented  for  Its  Interest  value,  as  examples  of  the  principles  or  rules  described  in 
the  goals,  or  as  context  materials  that  will  make  the  targeted  material  easier  to  understand. 

However,  students  sometimes  have  difficulty  In  distinguishing  between  Information  that  Is 
Important  for  them  to  master  and  material  that  Is  peripheral  to  the  central  objectives  of 
Instruction.  They  find  themselves  mastering  examples  and  context  material,  while  falling  to 
attend  to  the  most  Important  Information. 

The  most  direct  way  of  dealing  with  the  problem  of  student  uncertainty  about  what  to  focus  on 
Is  to  provide  students  with  the  Instructional  objectives  for  the  course.  In  general,  these 
objectives  should  be  either  the  same  as,  or  derived  from,  the  objectives  that  the  teacher  or 
Instructional  developer  began  with  prior  to  designing  the  Instructional  sequence. 

In  the  last  10  years,  there  has  been  a  considerable  amount  of  research  on  the  effects  of 
providing  students  with  Instructional  objectives.  This  research  began  with  a  study  by  Rothkopf 
and  Kaplan  (1972)  In  which  college  and  high  school  students  either  read  or  did  not  read  learning 
objectives  prior  to  studying  a  3,000-word  text  concerned  with  systems  training  at  a  telephone 
company.  They  found,  as  one  might  suspect,  that  students  who  received  objectives  learned  more  of 
the  material  specified  In  those  objectives  than  did  students  who  did  not  receive  objectives. 
Much  more  surprising,  however,  was  the  fact  that  the  group  that  received  objectives  also  learned 

more  of  the  material  that  was  not  keyed  to  an  objective.  This  result  suggested  that  providing 

students  with  learning  objectives  has  a  general  beneficial  effect  In  addition  to  the  specific 
effect  of  promoting  better  learning  of  material  identified  In  the  objectives.  The  results  of  the 
Rothkopf  and  Kaplan  (1972)  study  were  subsequently  replicated  and  expanded  In  studies  by  Kaplan 
(1976),  Kaplan  and  Rothkopf  (1974),  Kaplan  and  Simmons  (1974),  and  Nasslf-Royer  (1977).  In 
addition,  Ouchastel  and  Merrill  (1973)  have  provided  a  review  of  the  early  research  In  this  area. 

There  Is  little  doubt  that  providing  students  with  learning  objectives  can  have  positive 
effects.  There  Is  a  danger,  however,  that  teachers  should  be  aware  of.  Ernst  Rothkopf  (1968) 

once  suggested  that  many  students  operate  In  accordance  with  the  "law  of  least  effort."  That  Is, 

they  expend  only  the  amount  of  learning  effort  that  Is  required  to  attain  Instructional  goals. 
Teachers  also  can  be  guilty  of  operating  In  accordance  with  the  law  of  least  effort,  and  this  Is 
particularly  bad  when  they  generate  and  distribute  hastily  prepared  learning  objectives. 

It  is  very  easy  to  generate  trivial  learning  objectives.  You  simply  say  that  at  the  end  of 
Instruction,  students  will  know  such  and  such  facts.  The  danger  with  trivial  objectives  Is  that 
they  focus  attention  on  trivial,  low-level  knowledge  that  has  little  utility  outside  the 
classroom.  However,  if  teachers  expend  the  time  and  effort  required  to  generate  good  cognitive 
objectives,  the  students  will  benefit  from  knowing  what  aspects  of  Instruction  to  focus  on,  and 
the  Instruction  process,  as  a  whole,  will  benefit  from  a  focus  on  goals  that  are  truly  important. 

Asking  questions  during  Instruction.  Another  technique  for  maintaining  active  cognitive 
processing  Is  to  ask  students  questions  during  instruction.  We  can  see  when  students  are  looking 
at  a  book  or  watching  a  teacher,  but  we  cannot  see  whether  they  are  doing  anything  besides  merely 
gazing  at  the  book  or  teacher.  That  Is,  are  they,  in  fact,  processing  the  information  they  are 
receiving?  Asking  questions  is  one  way  of  Increasing  the  likelihood  that  a  student  is  actively 
attending  to  Instruction. 


Inserting  questions  In  written  material.  Several  years  ago,  Anderson  and  Biddle  (1975) 
conducted  an  extensive  review  of  the  research  literature  on  questioning  while  people  read;  they 
found  research  froa  early  In  this  century  (Gates,  1917)  that  showed  increased  learning  and 
retention  as  a  function  of  questioning.  In  the  last  15  years,  there  has  been  a  veritable 
explosion  In  research  on  this  topic,  and  Anderson  and  Biddle’s  (1975)  review  has  been  brought  up 
to  date  with  two  recent  reviews  by  Rickards  (1979)  and  Andre  (1979).  The  bulk  of  this  research 
can  be  traced  back  to  a  study  by  Rothkopf  (1966),  and  we  will  examine  this  experiment  in  detail 
as  an  exemplar  of  much  of  the  research  that  followed. 

Rothkopf  (1966)  had  students  read  a  5,200-word  text  concerned  with  oceanography,  under 
several  different  questioning  conditions.  Thirty-nine  objective,  completion-type  questions  were 
generated  from  this  text;  and  some  of  the  students  received  these  questions  prior  to  reading  the 
text.  Notice  that  this  procedure  would  allow  the  students  to  search  for  content  relevant  to 
answering  the  questions  they  had  just  seen.  Other  students  received  a  set  of  five  or  six 
questions  Imedlately  after  reading  the  segment  of  text  In  which  the  answers  to  the  questions 
could  be  found.  In  other  words,  this  group  knew  they  were  going  to  receive  questions,  but  they 
did  not  know  what  the  questions  were  until  they  had  finished  reading  the  relevant  segment  of 
text.  Two  final  groups  were  control  groups  that  read  the  text  after  having  been  Instructed  to 
learn  as  much  of  the  content  as  they  could. 

After  all  of  the  students  had  read  the  text,  a  test  was  administered  that  contained  two  kinds 
of  Information.  The  first  kind  was  Information  asked  for  In  the  questions  given  either  before  or 
after  reading  the  text.  The  second  kind  of  Information  was  Information  that  had  not  been  asked 
for  In  any  previous  questions.  As  one  might  expect,  students  who  had  received  questions  either 
before  or  after  reading  the  text  performed  better  on  the  same  questions  when  they  appeared  on  the 
test  than  did  students  who  had  not  received  the  questions.  Thus,  questioning  students  served  to 
Increase  the  learning  of  that  material  asked  for  In  the  questions.  The  second  major  outcome  of 
Rothkopf's  study  was  not  nearly  as  predictable.  The  students  who  received  questions  after 
reading  the  relevant  text  materials  recalled  more  of  the  material  not  tested  In  the  questions 
than  did  any  of  the  other  groups.  The  result  suggests  that  students  who  knew  they  were  going  to 
encounter  questions  after  reading  a  segment  of  text  attended  to  all  of  the  text  more  carefully 
than  students  In  most  of  the  other  groups,  thereby  Improving  their  performance  on  materials 
relevant  to  answering  questions  and  material  that  would  not  be  tested  until  the  final  post-test. 
In  other  words.  Inserting  questions  In  text  Increased  the  extent  to  which  that  text  was  actively 
processed  by  the  students. 

The  beneficial  effects  associated  with  embedding  questions  In  text  have  been  replicated  many 
times  since  Rothkopf's  (1966)  study.  For  Instance,  the  three  review  articles  previously 
mentioned  (Andre,  1979;  Anderson  &  81ddle,  1975;  Rickards,  1979)  examined  over  100  separate 
studies,  the  vast  majority  of  which  reported  results  consistent  with  Rothkopf's  (1966)  original 
study. 

Choosing  the  right  level  of  inserted  questions.  Another  important  concern  about  embedding 
questions  In  text  regards  the  level  at  which  the  questions  should  be  written.  Should  questions 
call  for  specific  factual  Information,  or  should  they  be  written  at  a  higher  level,  calling  for 
the  application  of  learned  Information  to  Instances  not  encountered  In  the  text?  Watts  and 
Anderson  (1971)  reported  a  study  that  addressed  this  Issue.  They  had  high  school  students  read  a 
text,  about  psychologists  and  their  theories,  under  four  question-type  conditions.  Two  groups 
read  the  passages  and  then  answered  questions  that  asked  for  a  low-level  factual  recall  or  the 
selection  of  an  example  of  a  principle  that  had  been  seen  before.  A  third  group  was  asked 
application  questions  that  called  for  applying  principles  to  examples  they  had  not  seen  before. 
The  final  group  read  the  passages  without  questions.  After  completing  the  study  phase  of  the 
experiment,  all  of  the  students  took  a  post-test  that  contained  factual  and  application 
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questions,  soae  of  which  had  been  seen  before  the  question  groups  and  some  of  which  were  new. 
The  results  of  the  study  were  that  the  students  receiving  application  questions  performed  better 
on  the  test  than  did  any  of  the  other  groups.  The  application  group  not  only  performed  at  a 
higher  level  on  application  questions,  but  they  also  performed  at  a  very  high  level  on  the 
remaining  types  of  questions. 

A  learning  strategy  approach  based  on  cognitive  theory.  Up  to  this  point,  we  have  focused  on 
Isolated  techniques  for  focusing  student  attention  and  maintaining  high  levels  of  cognitive 
processing.  Oonald  Dansereau  and  his  colleagues  recently  combined  a  variety  of  techniques  Into  a 
cognitive -based  learning  system  (Dansereau,  1978;  Dansereau,  Actklnson,  Long,  &  McDonald,  1974; 
Dansereau,  et  al.,  1979).  Dansereau's  system  Is  designed  to  teach  students  strategies  that  will 
enhance  the  learning  and  comprehension  of  Instruction  to  which  they  are  exposed. 

The  steps  In  Dansereau's  system  are  encoded  In  the  acronym  “MURDER."  The  M  stands  for 
establishing  an  appropriate  mood  for  study;  the  U,  for  promoting  understanding  which  Involves 
activities  such  as  marking  Important  and  different  Ideas;  the  R,  for  recalling  material  without 
referring  to  the  text;  the  0,  for  digesting  material  by  amplifying  It  and  storing  It;  the  E,  for 
expanding  knowledge  by  self-inquiry;  and  finally,  the  R,  for  reviewing  mistakes  by  learning  from 
tests. 

Each  of  the  steps  In  Dansereau's  system  Involves  a  number  of  substrategies.  For  example, 
substrategies  taught  to  students  under  the  recall  step  Include  paraphrasing  of  Incoming  text  (to 
transform  It  to  the  student's  own  words)  and  developing  Imagery  of  the  material  whenever  It  Is 
possible  to  do  so;  networking  of  text  material  that  Involves  linking  key  concepts  together  In  a 
manner  that  explicates  hierarchical  and  semantic  relationships  between  the  concepts;  and  finally, 
analysis  of  key  Ideas,  In  which  the  student  Identifies  key  Ideas  or  concepts,  develops  systematic 
definitions  and  elaborations  of  those  concepts,  and  then  Interrelates  the  Important  concepts. 
Dansereau  and  his  colleagues  have  developed  a  two-semester-hour  course  at  Texas  Christian 
University  that  teaches  students  to  utilize  the  strategies  In  his  study  system.  Students  who 
have  completed  this  course  have  performed  significantly  better  on  end-of-semester  learning 
exercises  than  did  students  who  did  not  take  the  course. 


Accessing  Prior  Knowledge 


The  constructivist  view  of  cognitive  learning  places  strong  emphasis  on  the  role  of  prior 
knowledge.  If  the  learner  does  not  have  prior  knowledge  relevant  to  the  construction  of  an 

Interpretation,  learning  via  memorization  may  occur,  but  understanding  will  not.  Accordingly, 

determining  whether  a  student  has  knowledge  relevant  to  the  Interpretation  of  the 
to-be-experienced  Instruction  or  whether  the  student  has  knowledge  that  can  be  activated  In  a 
mnemonic  activity  or  a  leamlng-by -analogy  activity  seems  of  considerable  Importance.  If  the 
student  has  relevant  background  knowledge,  the  preconditions  for  understanding  are  In  place  and 
no  special  educational  Intervention  Is  needed.  If  the  student  does  not  have  the  necessary 

background  knowledge  but  does  have  useful  knowledge  that  can  be  activated  using  mnemonics  or 

analogical  techniques,  these  techniques  can  be  Incorporated  In  the  Instructional  sequence.  In 
the  situation  where  the  student  does  not  have  relevant  background  knowledge  or  knowledge  that  can 
be  activated  to  support  mnemonic  or  analogical  activities,  the  students  must  be  taught  something 
new  that  can  bridge  the  gap  between  what  they  already  know  and  what  they  are  going  to  learn. 


It  should  be  emphasized  that  there  Is  a  distinction  between  assessing  specific  content 
knowledge  relevant  to  the  Interpretation  of  Instructional  material  and  assessing  aptitudes  that 
may  be  predictive  of  future  performance.  This  difference  relates  to  the  edumetrlc /psychometric 
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distinction  discussed  by  Carver  (1974)  and  to  the  different  purposes  of  comprehension  assessment 
discussed  by  Royer  and  Cunningham  (1981). 

One  of  the  major  Issues  facing  researchers  Interested  In  developing  Instruction  designed  to 
enhance  understanding  Is  how  to  devise  techniques  to  assess  knowledge  that  students  already  have 
and  that  will  be  relevant  to  the  Interpretation  and  acquisition  of  subsequently  encountered 
Instruction.  One  promising  way  of  doing  this  Is  the  Sentence  Verification  Technique  (SVT)  of 
measuring  comprehension,  developed  by  Royer.  The  technique  Involves  constructing  four  test 
sentences  for  every  original  sentence  that  appears  In  a  text.  The  first  kind  of  test  sentence  Is 
called  an  original  and  Is  the  exact  repetition  of  a  sentence  as  it  appeared  in  the  passage.  The 
second  type  of  test  sentence,  called  a  paraphrase,  is  developed  by  changing  as  many  words  as 
possible  In  an  original  sentence  without  altering  the  meaning.  The  third  type  of  test  sentence 
Is  called  a  meaning  change  and  Is  developed  by  changing  one  or  two  words  In  an  original  sentence 
In  a  manner  that  alters  the  meaning  of  the  sentence.  The  fourth  kind  of  test  sentence  Is  called 
a  dlstractor  and  Is  a  sentence  that  Is  related  to  the  theme  of  the  passage  but  unrelated  to  any 
sentence  appearing  In  the  passage. 

An  SVT  test  is  constructed  by  selecting  one  of  the  four  types  of  test  sentences  to  represent 
each  original  sentence  in  the  passage.  The  sentences  are  combined  such  that  each  test  sentence 
type  Is  equally  represented.  The  test  Is  administered  by  having  the  examinee  read  the  original 
passage  and  then  to  respond  "old"  or  "new"  to  each  of  the  test  sentences  (without  reexamining  the 
passage).  Old  sentences  are  defined  as  those  that  are  the  same  as  or  mean  the  same  as  original 
sentences  (originals  and  paraphrases),  and  new  sentences  are  defined  as  sentences  that  mean 
something  different  from  an  original  sentence  (meaning  changes  and  dlstractors). 

The  rationale  for  using  the  SVT  as  a  measure  of  comprehension  Is  based  on  the  assumption  that 
when  an  oral  or  written  message  Is  understood,  it  is  represented  in  memory  in  a  form  that 
preserves  the  meaning  of  what  was  read  or  heard  but  not  In  the  exact  words.  Hence,  subjects  who 
have  understood  the  message  should  be  able  to  correctly  identify  original  and  paraphrase 
sentences  as  measuring  the  same  thing  as  their  memory  representation,  and  to  correctly  reject 
meaning  changes  and  dlstractors  as  meaning  something  different  from  their  memory  representations. 

To  this  point,  Royer  and  his  associates  have  accumulated  considerable  evidence  supporting  the 
construct  validity  of  the  SVT  as  a  measure  of  both  written  and  oral  comprehension.  Briefly,  this 
evidence  shows  the  following:  (a)  The  SVT  Is  sensitive  to  text  readability  (Royer,  Hastings,  4 
Hook,  1979;  Royer,  Kulhavy,  Lee,  4  Peterson,  in  press);  (b)  the  SVT  is  sensitive  to  reading  skill 
as  defined  by  grade  level  (e.g.,  4th  versus  6th  graders)  and  by  teacher-defined  reading  groups 
(Rasool  4  Royer,  In  press;  Royer  et  al.,  1979;  Royer  4  Hambleton,  In  press;  Royer,  Kulhavy,  Lee, 
4  Peterson,  In  press);  (c)  the  SVT  is  sensitive  to  differences  in  subject-matter  expertise  when 
experts  or. non-experts  read  technical  text  (Royer,  Lynch,  Hambleton,  4  Bulgarelli,  in  press);  (d) 
the  SVT  Is  sensitive  to  Klntschian  text  characteristics  (Royer,  Lynch,  Hambleton,  4  Bulgarelli, 
in  press);  (e)  the  SVT  correlates  positively  with  standardized  reading  comprehension  test 
performance,  tests  of  intelligence,  and  several  other  measures  of  cognitive  performance  (Royer  et 
al.,  1979;  Royer,  Kulhavy,  Lee,  4  Peterson,  in  press);  (f)  the  SVT  measures  both  reading  and 
listening  comprehension  in  a  theoretically  consistent  manner  (Royer,  Kulhavy,  Lee,  4  Peterson,  in 
press);  and  (g)  the  SVT  is  a  highly  reliable  measure  of  comprehension  (Royer  4  Hambleton,  in 
press;  Royer,  Kulhavy,  Lee,  4  Peterson,  in  press). 

The  possibility  of  using  the  SVT  as  a  measure  of  specific  subject-matter  knowledge  stems  from 
another  line  of  research  undertaken  by  Royer  and  his  associates.  This  research  is  based  on  the 
previously  discussed  assumption  that  relevant  prior  knowledge  is  a  prerequisite  of  language 
comprehension.  Thus,  the  extent  to  which  someone  can  read  and  understand  a  text  based  on  subject 
matter  is  an  index  of  the  relevant  knowledge  the  person  has  about  the  subject  matter.  Moreover, 


this  leads  to  the  possibility  that  the  amount  of  knowledge  a  person  has  in  a  particular 
subject-matter  area  will  be  a  predictor  of  the  amount  of  knowledge  the  person  can  acquire  from  a 
course  of  study  In  that  content  area. 

Based  on  the  above  line  of  reasoning,  Royer  and  Marchant  have  Initiated  a  series  of  studies 
In  which  on  the  first  day  of  class,  students  enrolled  In  a  college-level.  Introductory  psychology 
course  read  abstracts  of  psychology  journal  articles  and  take  SVTs  based  on  those  articles.  We 
now  have  data  from  around  200  students  and  have  found  that  correlations  between  performance  on 
SVTs  administered  at  the  beginning  of  the  school  year  and  end-of-course  performance  was  .46.  In 
comparison,  the  correlations  between  Scholastic  Aptitude  Test  (SAT)  scores  and  end-of-course 
performance  was  .39. 

The  next  step  In  this  research  program  Is  to  expand  the  range  of  tests  that  are  examined. 
For  example,  we  Intend  to  replicate  the  predictive  studies  In  courses  Involving  the  natural 
sciences.  At  some  point,  we  would  also  like  to  conduct  studies  that  Involve  predicting  physical 
performance;  for  example,  measuring  comprehension  of  a  set  of  Instructions  describing  a  physical 
task  using  the  SVT,  and  then  using  actual  ability  to  perform  the  task  as  a  criterion  measure. 

The  SVT  procedure  for  measuring  comprehension  represents  only  one  way  to  attempt  to  Increase 
the  predictive  capability  of  Instrimients  used  to  select  personnel  for  specific  training 
experiences.  There  are  undoubtedly  many  ways  to  accomplish  this  end,  and  cognitive  theory  would 
suggest  that  any  procedure  that  assesses  the  extent  of  prior  knowledge  that  will  be  relevant  to  a 
subsequent  training  experience  would  be  useful  as  a  prediction  of  degree  of  success  In  training. 


An  ISP  Model  Designed  to  Produce  Understanding 

The  Ideas  presented  thus  far  In  the  paper  can  be  Incorporated  In  a  revision  of  the  familiar 
Instructional  Systems  Development  (ISO)  model.  The  revised  ISO  model  I  have  In  mind  Is  presented 
In  Figure  6-2.  It  should  be  noted  that  my  revised  model  represents  one  approach  to  developing 
Instruction  that  has  performance  validity,  as  Irv  Goldstein  has  used  that  term  In  his  paper.  I 
would  also  like  to  note  that  the  revised  model  I  have  In  mind  Is  not  meant  to  replace  the 
traditional  ISD  model,  but  Instead,  to  complement  It.  To  use  Irv's  taxonomy,  the  traditional  ISD 
model  represents  one  approach  to  training  validity,  whereas  my  revision  represents  one  approach 
to  performance  validity.  Both  are  obviously  critical  to  any  training  program.  Moreover,  I  think 
many  of  the  Ideas  present  In  Tom  Shuell's  paper  are  especially  relevant  to  developing  Instruction 
that  has  training  validity. 

Transfer  task  analysis  and  transfer  objectives.  The  model  begins  with  what  I  call  transfer 
task  analysis  and  the  development  of  transfer  objectives.  The  Idea  behind  transfer  task  analysis 
Is  to  analyze  the  domain  of  activities  a  trainee  might  engage  In  after  training.  In  an  effort  to 
Identify  types  of  tasks  that  are  represented  In  virtually  every  job  activity  a  trainee  might 
engage  In.  Those  activities  can  then  serve  as  indices  of  the  ability  of  training  to  transfer  to 
relevant  after-training  activities.  Based  on  what  I  have  read  In  Bob  Gagnd's  paper.  It  would 
appear  that  considerable  work  of  this  type  has  already  been  completed. 

Once  a  set  of  transfer  activities  has  been  Identified,  the  next  step  is  to  define  transfer 
objectives.  Transfer  objectives  can  be  thought  of  as  being  distributed  along  a  continuum  of  near 
transfer  to  far  transfer,  with  near-transfer  tasking  Involving  conditions  very  similar  to  those 
operative  during  training,  and  far  transfer  Involving  conditions  very  different  from  the 
Involved  In  training  (Royer,  1979).  As  an  example.  Imagine  a  training  sequence  directed  at  the 
repair  of  radio  receivers.  A  near-transfer  objective  might  ask  the  trainee  to  read  a  repair 
manual  on  a  receiver  similar  to  the  ones  used  In  training  and  then  take  a  comprehension  test  to 
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see  if  whet  was  read  was  understood;  or,  as  another  example,  the  trainee  might  be  expected  to 
perform  a  familiar  test  operation  on  a  receiver  that  had  never  been  seen  before,  but  was  similar 
to  those  used  in  training.  An  example  of  a  far-transfer  task  might  involve  asking  a  trainee  to 
design  and  construct  a  circuit  for  an  inoperable  receiver  from  raw  components.  This  might 
include  the  simulation  of  field  conditions  in  which  the  trainee  would  have  to  exhibit 

inventiveness  in  order  to  satisfy  the  objective. 

It  is  likely  that  most  transfer  objectives  are  going  to  fall  in  the  range  of  near-  to 
mid-transfer  level.  Satisfactory  performance  on  objectives  at  this  level  would,  I  think,  be 
consistent  with  what  Bob  Glaser  Imagines  an  expert  apprentice  to  be  able  to  do.  There  may, 
however,  be  situations  calling  for  the  capability  of  performing  far-transfer  activities.  For 

example,  training  of  special  forces  units  may  very  well  call  for  mastery  of  skills  falling  into 

the  far-transfer  range. 

Developing  and  trying  out  instruction.  The  next  phase  of  the  revised  ISO  model  would  entail 
developing  instruction  and  then  trying  It  out  In  a  small-scale  test.  Instructional  development 
at  this  point  would  not  involve  any  special  attempts  to  enhance  understanding.  After  the 

Instruction  was  developed.  It  could  be  tried  out  on  a  small  scale  to  see  if  the  transfer 
objectives  were  satisfied.  If  they  were,  no  further  instructional  development  would  be  necessary. 

Building  in  techniques  to  enhance  understanding.  If  performance  on  the  transfer  objectives 
is  not  satisfactory,  the  next  step  in  the  model  is  to  build-in  techniques  designed  to  enhance 
understanding.  The  first  stage  of  this  process  entails  determining  whether  trainees  have 
background  knowledge  on  which  to  base  an  approach  to  developing  understanding.  In  general,  this 
would  probably  be  done  Intuitively.  For  example,  if  the  Instructional  designer  has  an  analogy  In 
mind  that  might  enhance  understanding,  he  or  she  would  also  be  likely  to  have  a  good  sense  of 
whether  the  trainees  have  the  knowledge  on  which  the  analogy  is  based.  Alternatively,  if  there 
did  not  appear  to  be  a  knowledge  base  from  which  an  approach  to  understanding  could  be  developed, 
a  technique  to  establish  a  bridge  between  prior  knowledge  and  the  material  to  be  learned  would  be 
required. 

The  knowledge-enhancing  technique  that  was  chosen  would  depend  on  the  nature  of  the  material 
to  be  learned.  For  Instance,  if  the  training  Involved  the  acquisition  of  a  set  of  terms  or 
concepts.  It  would  be  appropriate  to  use  a  mnemonic  strategy.  Alternatively,  if  a  more  complex 
principle  or  rule  were  Involved,  analogies,  advanced  organizers,  or  concrete  models  might  be 
appropriate. 

Integrating  understanding  treatments  into  instruction.  After  an  approach  to  enhancing 
understanding  had  been  selected  and  instruction  developed,  the  final  stage  of  the  model  would 
entail  Integrating  the  new  material  into  the  previously  developed  instruction.  This  would  be 
followed  by  another  tryout  and  revision  cycle  until  the  transfer  objectives  had  been  satisfied. 


Research  Questions  for  the  Learning  Research  Laboratory 


In  my  paper  I  have  tried  to  present  what  I  perceive  to  be  a  developing  consensus  on  what 
understanding  is  and  how  It  might  be  operational ly  defined,  measured,  and  enhanced  via 
instruction.  Many  of  these  ideas  are  tentative  and  need  to  be  developed  further.  Below  are 
several  programs  of  research  that  would  contribute  to  the  development  process. 


Assessment  of  prior  knowledge  as  a  predictor  of  training  performance.  The  working  hypothesis 
here  is  that  assessment  of  knowledge  specifically  relevant  to  a  particular  course  of  instruction 
will  be  a  better  predictor  of  end-of-training  performance  than  will  more  general  aptitude 
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measures.  The  key  to  evaluating  this  hypothesis  Is  to  devise  Measurement  procedures  that  entail  » 

performance  of  some  activity  actually  Involved  during  training.  Examples  might  be  the  ability  to  * 

comprehend  text  samples  from  an  actual  training  manual  or  the  assessment  of  learning  from  a  j 

“mini -lesson"  derived  from  the  training  sequence.  | 

Transfer  objectives  as  Indices  of  understanding.  There  are  two  lines  of  research  I  will 
propose  In  this  area.  The  first  would  entail  the  development  of  a  methodology  for  developing  > 

transfer  objectives.  Part  of  this  would  Involve  developing  a  procedure  for  Isolating  tasks  that  J 

are  common  to  a  domain  of  work  activity.  The  next  part  would  entail  developing  a  procedure  for  j 

developing  transfer  objectives  from  the  Identified  tasks  common  to  the  domain.  This  would 

Involve,  for  example,  establishing  guidelines  for  the  selection  of  transfer  objectives  that  range 
along  the  near-to-far-transfer  dimension. 

The  second  line  of  research  In  this  area  would  be  the  evolution  of  the  hypothesis  that 
trainees  who  can  perform  at  a  satisfactory  level  on  transfer  objectives  at  the  end  of  training 
will  be  able  to  perform  better  on  the  job  than  will  trainees  who  did  not  attain  satisfactory 
performance  on  the  transfer  objectives.  In  short,  there  needs  to  be  research  on  whether  transfer  i 

objectives  can  serve  as  the  measurement  base  for  the  performance  validity  of  a  training  sequence. 

i 

Research  on  techniques  designed  to  enhance  understanding.  Earlier  In  the  paper,  four  1 

techniques  were  discussed  that  either  activated  relevant  prior  knowledge  or  that  established  a  j 

bridge  between  prior  knowledge  and  material  to  be  learned.  Whereas  there  Is  much  literature 
establishing  the  utility  of  these  techniques  In  a  variety  of  educational  situations,  I  know  of  no 
research  that  examines  the  utility  of  the  techniques  In  military  training  situations.  I  think 
three  of  the  techniques  are  especially  promising.  Mnemonic  techniques  similar  to  the  keyword 
method  have  been  shown  to  be  extremely  powerful  techniques  for  learning  a  variety  of  materials, 
and  their  utility  for  Air  Force  training  problems  should  be  examined.  I  think  the  use  of 
analogies  for  Instructional  purposes  Is  just  beginning  to  be  examined  In  detail,  and  their  use 
holds  a  great  deal  of  promise  for  developing  Instruction  that  enhances  understanding.  Concrete 
models  are  also  very  promising.  Rich  Mayer's  research  has  already  shown  the  effectiveness  of 
such  models,  and  I  would  anticipate  positive  effects  could  also  be  shown  In  military  training 
situations.  I  am  less  optimistic  about  advance  organizers,  even  though  there  Is  substantial 
evidence  showing  their  effectiveness.  These  effects  are,  however,  generally  small,  and  there  Is 
still  substantial  doubt  about  what  an  advance  organizer  Is. 
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CHAPTER  7 


THE  RELATIONSHIP  OF  TRAINING  GOALS  TO 
BASIC  RESEARCH  ISSUES  IN  INSTRUCTION 


Irwin  L.  Goldstein 
University  of  Maryland 

Instructional  systems  philosophy  as  developed  by  Gagni  and  his  colleagues  (e.g.,  Briggs, 
Campeau,  Gagn6,  S  May,  1967;  Gagnd,  1970)  has  had  a  profound  Impact  on  the  way  that  such  systems 
are  conceptualized.  Similar  developments  stimulated  by  these  Instructional  theorists  have 
affected  the  design  of  training  systems  In  both  the  public  and  private  sectors.  These  approaches 
(e.g.,  Goldstein,  1974,  1978)  typically  emphasize  the  specification  of  Instructional  objectives 
based  on  needs  assessment  procedures,  precisely  controlled  learning  experiences  to  achieve  these 
objectives,  criteria  for  performance,  and  evaluation  Information.  In  a  training  framework.  It  Is 
possible  to  examine  the  components  of  Instructional  systems  models  and  ask  what  the  user  desires 
to  achieve;  l.e.,  "What  are  the  goals  of  the  Instructional  system?*  An  examination  of  those 
goals  and  of  the  supporting  Instructional  system  components  necessary  to  achieve  them  suggests  a 
number  of  areas  that  require  further  research  and  study.  This  paper  examines  these  goals,  the 
components  of  Instructional  systems,  and  some  suggested  research  Issues. 


Goals  of  Instructional  Programs 


As  presented  In  Figure  7-1,  there  are  four  potential  goals  for  training  systems: 

1.  Training  validity.  This  particular  goal  refers  only  to  the  validity  of  the  training 
program.  Validity  Is  determined  by  the  performance  of  trainees  on  criteria  established  for  the 
training  program. 

2.  Performance  validity.  This  goal  refers  to  the  validity  of  the  training  program  as 
measured  by  performance  In  the  transfer  of  what  has  been  learned  to  the  job  setting. 

3.  Intra-organlzatlonal  validity.  This  concept  refers  to  the  performance  of  a  new  group  of 
trainees  within  the  same  organization  for  which  the  original  training  program  was  developed.  In 
this  instance,  the  question  is  whether  generalization  will  occur  to  the  performance  of  new 
trainees  based  on  the  evaluated  performance  of  a  previous  group. 

4.  Inter-organizational  validity.  In  this  Instance,  the  goal  Is  to  determine  whether  a 
training  program  validated  In  one  organization  can  be  utilized  In  another  organization. 

A  consideration  of  these  categories  Indicates  that  the  achievement  of  validity  at  each 
succeeding  stage  Is  affected  by  an  Increasing  number  of  variables.  For  example,  the 
establishment  of  training  validity  requires  the  consideration  of  evaluation  procedures.  Including 
criteria  of  training  success.  Trainee  performance  on  the  job  (performance  validity)  not  only 
requires  the  consideration  of  training  criteria  of  success  but  also  demands  an  examination  of  the 
potential  disruptive  effects  of  organizational  constraints.  For  example,  the  training  program 
and  the  organization  often  specify  different  performance  objectives,  and  trainees  are  caught  In  a 
conflict  which  sometimes  results  In  failure.  As  a  result,  the  trainee  and  training  program  may 
be  declared  Inadequate  because  the  training  analyst  has  not  considered  the  relevant  variables 
which  determine  success  or  failure  at  each  of  the  four  stages  of  validity.  A  consideration  of 
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Figure  7-1 .  The  Relationship  of  Training  System  Goals  and  Instructional  Design  Components. 


67 


each  of  these  goals  and  their  related  Instructional  processes  Is  presented  next.  After  a 
discussion  of  these  Issues,  some  research  program  Issues  are  suggested. 


As  noted  above,  training  validity  refers  to  the  establishment  of  treatment  effects  as  a 
result  of  the  Instructional  program.  In  this  situation,  the  pursuit  of  validity  refers  only  to 
performance  In  the  Instructional  program  and  does  not  consider  transfer  performance.  While 
training  analysts  might  note  that  what  occurs  In  the  training  program  does  not  provide  any 
guarantees  about  learning  transfer  to  the  job.  It  should  be  obvious  that  the  vast  majority  of 
Instructional  programs  attend  only  to  training  performance  and  not  to  transfer  performance.  Of 
course,  many  would  argue  that  even  when  training  programs  are  not  designed  on  the  basis  of  a 
needs  assessment  of  the  job,  there  Is  still  some  mental  appreciation  of  the  transfer  performance 
around  which  the  Instructional  program  Is  designed.  For  example,  there  are  many  managerial 
training  programs  that  have  as  a  goal  Instructing  trainees  In  the  basic  theory  of  management. 
Evidently,  managers  are  expected  to  return  to  their  array  of  jobs  and  situations  with  what  they 
have  learned  and  are  to  determine  a  way  to  apply  It.  If  one  questions  the  developers  of  such 
programs,  they  will  readily  admit  that  they  have  not  done  a  needs  assessment  and  have  not 
measured  transfer  performance,  but  somehow  or  other  they  have  a  *feel“  for  the  job  setting  and 
thus  have  designed  their  program  appropriately.  Evidence  for  this  viewpoint  Is  nonexistent.  It 
Is  necessary  to  simply  accept  the  fact  that  this  type  of  training  program  Is  concerned  only  with 
training  performance;  It  Is  focusing  only  on  the  establishment  of  training  validity.  As  we  shall 
see,  that  task  In  Itself  Is  not  simple. 

It  Is  interesting  to  note  that  some  programs  are  primarily  designed  to  achieve  training 
validity.  Thus,  Instructional  programs  In  academic  environments  are  often  designed  to  teach  a 
particular  subject  matter  primarily  because  of  the  value  of  the  content  and  not  necessarily 
because  the  learning  Is  expected  to  transfer  to  other  specific  settings.  Also,  the  purpose  of 
the  Implementation  of  some  training  programs  is  to  provide  a  research  vehicle.  For  example,  an 
Investigator  might  be  concerned  with  the  effects  of  different  types  of  feedback  on  the 
performance  of  trainees  learning  a  complex  motor  skill.  In  this  case,  the  program  Is  concerned 
primarily  with  training  validity,  not  necessarily  performance  validity.  It  Is  possible  to  debate 
whether  or  not  training  programs  should  be  permitted  to  focus  only  on  training  validity  without  a 
concern  for  the  transfer  setting.  However,  that  value  judgement  Is  not  the  focal  point  of  this 
discussion.  Rather,  the  point  here  Is  that  such  programs  do  exist  and  that  the  establishment  of 
training  validity  In  Itself  requires  the  consideration  of  many  Important  variables.  Some  of  the 
most  Important  points  to  consider  Include  the  following. 


Needs  Assessment  for  Training  Validity 


First,  It  Is  Important  to  realize  that  some  form  of  needs  assessment  Is  necessary.  In  the 
case  of  establishing  training  validity,  the  needs  assessment  would  not  necessarily  focus  on  the 
behaviors  required  In  the  transfer  setting.  Rather,  the  needs  assessment  would  attend  to  the 
particular  goals  and  objectives  to  be  achieved  as  a  result  of  the  Instructional  program.  Thus, 
If  the  program  Is  designed  to  provide  a  working  knowledge  of  management  philosophy.  It  is 
necessary  to  determine  through  task  analysis  and  specification  of  Instructional  objectives  what 
particular  goals  the  training  program  Is  trying  to  achieve.  It  is  also  Important  to  perform  a 
person  analysis  so  that  the  Instructional  program  Is  designed  to  fit  the  knowledge,  skill  and 
ability  characteristics  of  the  trainee  population.  Indeed,  this  needs  assessment  must  provide 
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the  Input  for  the  design  of  the  Instructional  sequences,  as  well  as  the  suggested  criteria  by 
which  the  training  program  will  be  evaluated.  It  Is  not  reasonable  to  skip  the  needs  assessment 
process  just  because  what  Is  being  planned  Is  a  generalized  training  program  that  Is  not  geared 
to  specific  behaviors  to  be  performed  In  a  transfer  setting.  There  should  still  be  performance 
objectives,  designed  for  particular  trainee  populations,  that  are  determined  by  needs  assessment 
procedures.  Unfortunately,  Instead  of  careful  needs  assessment,  the  training  field  often  appears 
dominated  by  a  fads  approach.  Analysts  seldom  find  out  very  much  about  their  particular  approach 
before  they  are  off  examining  another  type  of  program.  The  fads  approach  places  a  heavy  emphasis 
on  the  development  of  techniques,  without  consideration  of  needs  assessment  followed  by  a 
matching  of  the  techniques  to  the  needs.  Training  analysts  still  have  to  be  warned  about 
selecting  tools  and  finding  something  they  fit,  by  quotes  like  the  following: 

If  you  don't  have  a  gadget  called  a  teaching  machine,  don't  get  one.  Don't  buy  one; 
don't  borrow  one;  don't  steal  one.  If  you  have  such  a  gadget,  get  rid  of  It.  Don't 
give  It  away,  for  someone  else  might  use  It.  This  Is  a  most  practical  rule  based  on 
empirical  facts  from  considerable  observation.  If  you  begin  with  a  device  of  any 
kind,  you  will  try  to  develop  the  teaching  program  to  fit  that  device.  (Gilbert, 

1960,  p.  478). 

Gilbert  Is  not  saying  that  teaching  machines,  or  sensitivity  training,  or  computer-assisted 
Instruction  (CAI),  or  any  other  technique  does  not  work.  He  Is  saying  that  the  design  of  change 
programs  cannot  begin  with  Instructional  media.  Instead,  one  must,  through  needs  assessment, 
determine  the  objectives  of  training  so  that  the  criteria  for  evaluation  and  the  choice  of 
instructional  program  are  based  on  sound  decisions.  It  appears  to  be  unfortunate  but  true  that 
programs  that  are  designed  to  achieve  only  training  validity  are  also  more  likely  to  be  designed 
on  the  basis  of  the  most  popular  fad  rather  than  careful  needs  assessment. 


Research  Issues  for  Needs  Assessment  and  Training  Validity 


Interestingly,  even  the  apparently  simple  Issues  concerning  needs  assessment  for  training 
validity  raise  some  very  complex  Issues.  Training  analysts  remain  uncertain  about  what  the 
appropriate  Input  should  be  for  the  design  of  Instructional  programs.  Needs  assessment  of  jobs 
has  focused  on  a  number  of  different  factors  Including  the  determination  of  the  tasks  to  be 
performed;  the  knowledge,  skills  and  abilities  necessary  to  perform  the  tasks;  and  the  behaviors 
necessary  to  demonstrate  competent  performance.  However,  there  Is  virtually  no  Information 
available  comparing  the  various  systems  as  Inputs  to  the  design  of  training  systems.  There  is 
not  even  much  Information  on  the  best  systems  to  determine  which  characteristics  (e.g., 
frequency,  criticality)  should  be  measured  to  provide  Information  about:  tasks,  knowledge, 
skills  or  abilities,  or  performance.  The  opportunity  to  perform  needs  assessment  on  relevant 
jobs  and  to  examine  the  effects  of  the  system  utilized,  on  Instructional  design  and  learner 
performance,  is  a  unique  opportunity  for  a  basic  learning  laboratory. 


Criteria  and  Training  Validity 


In  general,  evaluation,  as  described  by  Goldstein  and  Buxton  (1982),  is  considered  to  be  an 
information-gathering  process  that  should  not  be  expected  to  lead  to  decisions  that  declare  a 
program  as  all  good  or  all  poor.  One  of  the  purposes  of  the  needs  assessment  Is  the 
determination  of  the  knowledge,  skills,  and  abilities  (KSAs)  required  for  successful  job 
performance.  As  shown  In  the  model  In  Figure  7-1,  that  Information  must  provide  direct  input  to 
the  training  program  to  determine  the  actual  content  of  the  Instructional  material.  The  same 
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Information  concerning  the  KSAs  necessary  for  successful  job  performance  should  also  provide  the 
Input  for  the  establishment  of  measures  of  training  success.  Logically,  we  should  want  our 
training  program  to  consist  of  the  materials  necessary  to  develop  the  KSAs  to  perform 
successfully  on  the  job.  Just  as  logically,  we  should  determine  the  success  of  our  training 
program  by  developing  measures  (or  criteria)  that  tell  the  training  evaluator  how  well  the 
training  program  does  In  teaching  the  trainees  the  same  KSAs  necessary  for  job  success.  Also, 
these  criteria  are  used  at  the  end  of  the  training  program  to  determine  how  well  our  program  Is 
doing  at  that  time;  and  as  discussed  In  the  section  on  transfer  validity,  these  criteria  should 
also  be  used  later  on,  when  the  trainees  are  on  the  job,  to  determine  how  much  of  the  KSAs 
learned  In  training  transferred  to  the  actual  job. 


Research  Issues  Concerning  Criteria  for  Training  Validity 


In  general,  investigations  of  the  types  of  criteria  employed  to  provide  evaluation  data  are 
not  reassuring.  For  example,  Catalanello  and  Kirkpatrick  (1968)  found  that  of  154  companies 
surveyed,  the  largest  number  (77S)  stressed  studies  related  to  reports  of  personal  reactions. 
Even  in  those  Instances  where  such  data  are  collected,  some  Investigators  (Mlndak  A  Anderson, 
1971)  have  suggested  that  most  of  these  measures  were  "eyeball"  attempts  to  measure  reactions. 
Few  studies  have  bothered  to  measure  learning  and  on-the-job  behavior.  It  is  probably  not 
unreasonable  to  suspect  that  these  Investigations  did  not  consider  criterion  relevance  and 
reliability.  The  question  about  how  criterion  relevance  is  determined  Is  not  easily  answered. 
There  should  be  a  tracing-out  process  from  the  needs  assessment  to  the  development  of  criteria. 
The  rules  of  this  process  are  unknown.  Indeed,  no  one  Is  really  very  sure  what  the  Input  should 
be  to  maximize  criterion  relevance.  Recently,  there  has  been  Increased  attention  to 
criterion-referenced  measures  employing  an  absolute  standard  of  quality  (Swezey,  1978),  rather 
than  norm-referenced  measures  that  are  dependent  on  a  relative  standard.  Criterion-referenced 
measures  provide  a  standard  of  achievement  for  the  individual  based  on  specific  behavior 

objectives  and  thereby  provide  an  Indication  of  the  degree  of  competence  attained  by  the 
trainee.  The  characteristics  of  criterion-referenced  measures  -  Including  methods  of  determining 
content  validity  as  well  as  the  design  of  methods  to  determine  reliability  -  need  attention. 

There  Is  also  great  difficulty  In  determining  what  constitutes  Ideal  performance  and  how  it 
should  be  measured.  Thus,  Wllllges,  Roscoe,  and  Willi ges  (1972)  note  that  there  is  little 

agreement  on  what  constitutes  Ideal  pilot  performance,  and  the  reliability  of  most 

pilot-performance  grading  systems  has  been  disappointing.  Consequently,  even  studies  of  original 
learning  are  difficult  to  complete  because  there  is  little  agreement  on  what  constitutes  terminal 
performance.  Even  In  the  area  of  simulators,  where  there  have  been  extensive  efforts,  the 
establishment  of  appropriate  criteria  for  complex  performance  is  a  serious  issue  demanding  the 
attention  of  researchers.  In  addition,  there  are  continuing  questions  concerning  the  development 
of  new  criterion  measures  that  provide  further  information  about  the  Instruction  process.  Thus, 
reaction  measures  which  only  ask  the  trainee  for  his  general  feelings  about  learning  have  not 
been  found  to  be  particularly  useful.  However,  it  Is  possible  that  trainees  are  able  to 

self-assess  their  learning  of  particular  components  of  KSAs.  Research  is  needed  to  determine 
whether  learners  can  assess  their  own  capabilities  concerning  how  much  they  have  learned.  There 
are  also  questions  concerning  the  relationships  between  these  self-assessment  indices  and  other 
measures  of  learning  in  the  instructional  environment. 


Evaluation  Models  and  Training  Validity 

Instructional  programs  are  never  complete;  instead,  they  are  designed  to  be  revised  on  the 
basis  of  information  obtained  from  evaluations  founded  on  relevant  multiple  criteria  that  are 
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reliable  and  free  from  contamination.  The  better  experimental  procedures  control  more  of  the 
potentially  disruptive  variables,  thereby  permitting  a  greater  degree  of  confidence  In  specifying 
program  effects.  Although  the  constraints  of  the  training  envlrormerrt  may  make  laboratory-type 
evaluations  lojiosslble  to  achieve,  an  awareness  of  the  Important  factors  In  experimental  design 
makes  It  possible  to  conduct  useful  evaluations  even  under  adverse  conditions. 

There  are  several  types  of  evaluation  In  models  employed  to  assess  learning.  Perhaps,  the 
best-known  model  Is  typically  titled  an  experimental  or  a  quasi -experimental  design  model.  These 
types  of  models,  which  are  sometimes  employed  In  evaluating  training  validity,  Include 
considerations  related  to  pre-  and  post-testing,  control  groups,  random  assignment  of  subjects, 
etc.  The  job  of  the  analyst  Is  to  choose  the  most  rigorous  design  possible  and  to  be  aware  of 
Its  limitations.  Unfortunately,  there  are  few  studies  that  employ  the  most  rigorous  design 
within  the  limitations  Imposed  by  the  environment.  In  many  Instances,  the  authors  of  research 
articles  apologize  In  advance  for  poor  design  that  obviously  could  have  been  strengthened  by  some 
Initial  effort.  It  appears  that  most  efforts  at  evaluation  are  afterthoughts.  Appropriate 
planning  and  the  use  of  elementary  research  procedures  would  significantly  Increase  the  amount  of 
valid  Information  gathered. 

Recently,  however,  training  researchers  have  begun  to  consider  other  types  of  evaluation 
models.  One  that  Is  now  being  used  quite  frequently  Is  an  Individual  difference  model.  In  this 
model,  learning  scores  are  correlated  with  later,  on-the-job  performance  scores  to  determine 
whether  the  same  pattern  of  Individual  differences  exists.  Thus,  If  persons  who  perform  well  In 
training  also  perform  well  on  the  job,  positive  correlations  would  result.  The  other  model  Is  a 
content  validity  model  where  the  analysis  examines  whether  the  KSAs  that  are  judged  critical  for 
job  performance  are  also  those  emphasized  In  training.  Further  discussion  of  these  models  will 
be  presented  In  the  discussion  on  transfer  performance  since  they  all  depend  on  transfer  data  as 
well  as  on  training  data.  They  are  presented  here  only  because  they  are  also  dependent  on 
training  validity  data.  This  also  Is  an  appropriate  place  to  hint  that  Issues  concerning  the 
Interrelationship  among  these  different  types  of  evaluation  models  pose  Important  research 
questions. 

Still  another  evaluation  model  In  training  validity  Is  called  a  process  model.  This  model  Is 
actually  a  partner  to  all  evaluation  models,  rather  than  a  completely  separate  way  of  examining 
the  world  of  training  validity.  There  Is  a  growing  awareness  that  the  use  of  evaluation  designs 
has  not  provided  the  degree  of  understanding  that  was  originally  anticipated.  The  evaluation 
designs  and  specification  of  outcome  criteria  have  been  based  on  a  product  or  outcome  view  of 
training  validity.  Thus,  researchers  collected  pre-  and  post-measures,  compared  them  with 
control  groups  and  discovered  that  they  did  not  understand  the  results  they  had  obtained.  This 
problem  was  especially  apparent  when  the  collectors  of  these  data  were  outside  consultants  who 
appeared  only  to  collect  pre-  and  post-data  but  had  no  conception  of  the  process  that  had 
occurred  In  training  between  the  pre-  and  post-measurement.  An  event  experienced  by  this  author 
Illustrates  this  Issue.  In  a  study  of  computer-assisted  Instruction  In  a  school  setting 
(Goldstein  t  Rosenberg,  1977),  two  teachers  each  agreed  to  Instruct  a  geometry  class  by 
traditional  methodology  and  by  CAI.  Thus,  each  teacher  taught  one  traditional  and  one  CAI 
class.  Further,  the  teachers  agreed  to  work  together  to  design  an  exam  that  would  cover  material 
presented  In  both  classes.  At  the  end  of  the  first  testing  period,  the  traditional  classes 
taught  by  each  teacher  significantly  outperformed  the  CAI  groups  taught  by  these  same  teachers. 
The  same  outcome  occurred  at  the  second  testing.  However,  at  the  third  testing,  one  of  the  CAI 
groups  Improved  so  that  It  was  now  equivalent  to  the  two  traditional  groups.  The  other  CAI  group 
performed  significantly  worse  than  the  other  three  Instructional  groups.  One  reasonable 
conclusion  for  this  series  of  events  Is  that  one  of  the  teachers  finally  learned  how  to  Instruct 
the  CAI  group  so  that  It  was  now  equivalent  to  the  two  traditional  groups,  whereas  the  other 
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teacher  had  not  been  able  to  perform  that  task  with  his  CAI  group.  Indeed,  If  the  Investigators 
had  collected  only  the  outcome  measures,  any  number  of  similar  erroneous  conclusions  could 
probably  have  been  offered  as  explanations  for  the  data.  In  this  case,  the  Investigators  also 
observed  the  Instructional  process,  to  provide  information  about  the  program.  The  evaluators  had 
learned  that  the  Instructor  for  the  CAI  group  that  eventually  Improved  had  become  disturbed  over 
the  performance  of  his  students.  Thus,  the  teacher  offered  remedial  tutoring  and  essentially 
turned  his  CAI  class  Into  a  traditional  group. 

Many  researchers  have  become  concerned  about  the  lack  of  understanding  about  the  variables  In 
the  training  process  which  affect  or  determine  outcomes.  The  view  Is  especially  well  expressed 
by  Cronbach  (1 973) : 

Insofar  as  possible,  evaluation  should  be  used  to  understand  how  the  course  produces 
Its  effects  and  what  parameters  Influence  Its  effectiveness.  It  is  Important  to 
learn,  for  example,  that  the  outcome  of  a  programed  Instruction  depends  very  much 
upon  the  attitude  of  the  teacher;  Indeed,  this  may  be  more  Important  than  to  learn 
that  on  the  average  such  Instruction  produces  slightly  better  or  worse  results  than 
conventional  programs,  (p.  675) 

An  understanding  of  these  process  Issues  -  which  are  the  ones  on  which  to  collect  data  -  and 
how  they  interact  with  learner  performance  are  all  important  research  Issues  that  deserve  further 
attention. 


Performance  Validity 


As  noted  earlier,  the  establishment  of  performance  validity  has  the  additional  burden  of 
determining  whether  performance  has  positively  transferred  from  the  training  program  to  the 
on-the-job  environment.  For  this  type  of  validity,  transfer  of  performance  Is  considered  only 
for  the  original  group  being  trained  In  the  Instructional  program.  The  question  of  generalizing 
to  new  training  populations  or  new  organizations  will  be  treated  later  In  this  paper. 
Considering  the  points  presented  In  the  discussion  of  training  validity.  It  may  appear  that  there 
are  relatively  few  additional  Issues  to  be  added  to  the  analysis  of  performance  validity. 
Unfortunately,  that  Is  not  the  truth.  Now  It  Is  necessary  to  consider  conditions  which  Involve 
the  transfer  of  performance  In  one  environment  (training)  to  another  environment  (on  the  job). 
This  perspective  has  Important  implications  for  the  needs  assessment  process,  because  now  the 
objectives  and  goals  of  the  Instructional  program  must  be  determined  by  a  needs  assessment  of  the 
transfer  environment. 


Needs  Assessment  and  Performance  Validity 

In  the  preceding  section  on  training  validity.  It  was  noted  that  the  needs  assessment  process 
was  still  necessary.  However,  It  was  limited  to  task  and  person  analysis  to  establish  learning 
objectives  that  were  not  necessarily  related  to  transfer  performance.  In  the  establishment  of 
performance  validity,  the  concern  Is  with  designing  training  programs  based  on  a  needs  assessment 
of  on-the-job  performance. 

There  are  dozens  of  studies  that  can  be  cited  to  demonstrate  the  dangers  of  designing 
training  programs  without  this  type  of  needs  assessment.  Perhaps  one  of  the  sadder  examples  Is 
revealed  by  Miller  and  Zeller  (1967)  In  an  Investigation  of  418  hard-core  unemployed  trainees  In 
a  program  to  train  highway  construction  machinery  operators.  The  authors  were  able  to  obtain 
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Information  from  270  graduates.  Of  this  group,  61%  of  the  graduates  were  employed  and  39%  were 
unemployed  at  the  time  of  the  Interview.  In  addition,  more  than  half  of  the  employed  group  said 
they  were  without  jobs  more  than  60%  of  the  time.  Some  of  the  reasons  for  the  unemployment 
situation  were  Inadequacies  In  training,  such  as  not  enough  task  practice.  The  details  showed 
that  the  program  was  not  based  on  a  consideration  of  job  components.  One  trainee  noted  that 
"contractors  laughed  when  I  showed  them  my  training  diploma  and  said  come  back  after  you  get  some 
schooling,  buddy"  (p.  32-33).  In  a  familiar  lament,  the  authors  of  the  report  wonder  how  a 
training  program  could  be  designed  without  a  thorough  analysis  of  the  tasks  and  the  KSAs  required. 

As  far  as  research  needs  concerning  traditional  needs  assessment,  they  are  all  specified  In 
the  previous  section  on  training  validity.  However,  there  Is  a  growing  realization  that  training 
occurs  In  one  organization  (training  organization)  and  performance  occurs  In  another  organization 
(the  work  organization).  Thus,  performance  validity  forces  consideration  of  the  fact  that 
something  learned  In  one  environment  (training)  will  be  performed  In  another  environment  (on  the 
job).  The  trainee  will  enter  a  new  environment,  to  be  affected  by  all  of  the  Interacting 
components  that  represent  organizations  today.  Certainly,  there  are  some  aspects  of  the 
environment  which  help  determine  the  success  or  failure  of  training  programs  beyond  the 
attributes  the  trainee  must  gain  as  a  result  of  attending  the  Instructional  program. 

The  actual  components  of  an  organizational  analysis  are  dependent  on  the  type  of  program 
being  Instituted  and  the  characteristics  of  the  organization  to  which  transfer  will  occur. 
However,  the  following  broad  categories  should  be  considered. 


1.  The  Specification  of  Goals 

When  organizational  goals  are  not  considered  In  the  Implementation  of  training  programs, 
objectives  and  criteria  that  ensue  from  the  needs  assessment  process  are  not  evaluated.  Later, 
the  organizations  are  not  able  to  specify  their  achievements  because  they  have  not  collected  the 
necessary  Information.  Clearly,  If  the  goals  are  not  specified.  It  Is  not  very  likely  that  they 
are  Included  In  the  design  of  the  Instructional  program. 

Organizations  might  expect  training  programs  to  provide  trainees  with  expectations  about  the 
job,  or  particular  views  toward  performance  requirements.  It  is  not  unusual,  for  example,  to 
discover  that  police  training  programs  are  devoted  to  skill  requirements  (e.g.,  operating  a 
police  vehicle  or  utilizing  firearms)  or  Information  requirements  (knowing  the  difference  between 
a  felony  and  misdemeanor).  Yet,  organizational  analyses  often  discover  that  there  are 
organizational  expectations  concerning  Interpersonal  relationships  with  the  public  and  a  concern 
for  all  citizens  regardless  of  race,  color,  or  creed.  Obviously,  If  these  organizational 
philosophies  and  emphases  are  not  clearly  defined  and  specified  during  the  needs  assessment 
phase,  they  will  not  be  considered  In  the  design  of  the  training  program  and,  at  best,  will 
receive  only  passing  attention  in  the  Instructional  sequence.  In  this  case,  the  problem  is 
further  complicated  by  the  fact  that  It  Is  a  lot  easier  to  teach  weapons  procedure  than  to 
instruct  In  the  topic  of  interpersonal  relations.  Thus,  certain  topics  may  be  de-emphaslzed  and 
sometimes  organizations  become  aware  that  they  have  a  problem  only  when  they  are  facing  a  series 
of  complaints.  At  that  point  In  time,  everyone  wonders  why  the  organizational  objectives  were 
not  translated  Into  training  and  job  requirements.  In  short,  many  training  programs  are  based  on 
teaching  skills  requirements  rather  than  on  unspecified  and  unidentified  organizational 
objectives.  Yet,  paradoxically,  organizations  often  judge  the  value  of  a  program  on  the  basis  of 
their  own  objectives  that  were  never  specified,  never  considered  in  the  design  of  the  program, 
and  never  utilized  in  designing  the  evaluation  model.  Thus,  the  financial  cost  of  some 
instructional  programs  resu'ts  In  their  early  demise  because  the  instructional  system  analysis 
did  not  Identify  that  variable  as  an  organizational  objective. 


2.  The  Determination  of  the  Organizational  Training  Climate 

As  complex  as  the  specification  of  system-wide  organizational  objectives  appear  to  be,  the 
determination  of  the  objectives  by  themselves  will  not  do  the  job.  Unfortunately,  many  of  the 
situations  are  also  marked  by  organizational  conflicts,  which  are  very  disruptive.  For  example, 
conflicts  between  Government  sponsors  of  a  program,  employers,  and  training  Institutions  can 
completely  disrupt  programs  (Goodman,  1969).  Many  of  these  conflicts  are  based  on  the  different 
parties  to  the  program  having  different  sets  of  equally  unspecified  goals  and  expectations.  A 
study  by  Salinger  (1973)  characterized  the  negative  consequences  for  a  poor  climate.  In  the 
study,  top  management  In  Federal  agencies  had  a  generally  negative  view  of  training  and  no  direct 
knowledge  of  Its  benefits.  This  resulted  In  a  system  that  failed  to  reward  managers  for 
effective  training  efforts.  Rather,  managers  failed  to  plan  or  budget  for  training.  It  Is  not 
difficult  to  guess  that  whatever  training  was  provided  did  not  serve  much  of  a  purpose  for  the 
Individual  or  the  organization.  Even  more  likely,  there  are  Insidious  conflicts  that  somehow 
(with  a  magic  wand)  the  training  program  Is  supposed  to  resolve.  Even  assuming  that  the  training 
program  could  be  appropriately  designed  In  such  a  situation,  It  Is  likely  It  will  succumb  to  the 
conflict. 


3.  The  Identification  of  Relevant  External  System  Factors 

The  preceding  sections  on  organizational  analysis  have  Identified  Issues  related  to  the 
failure  to  specify  organizational  goals  and  the  problems  of  organizational  conflicts.  The  Issue 
examined  In  this  section  Is  the  failure  to  recognize  the  Importance  of  the  Interacting 
constraints  acting  on  an  organization  and  their  effects  upon  training  programs.  These 
considerations  could  be  treated  as  a  failure  to  specify  organizational  goals  or  as  organizational 
conflicts.  However,  external  constraints  are  becoming  a  very  serious  problem  In  the  design  of 
training  programs.  Thus,  they  are  treated  as  a  separate  section.  The  design  of  Instructional 
programs  Is  affected  by  legal,  social,  economic,  and  political  factors.  Interrelationships  of 
these  variables  should  be  carefully  specified  during  organizational  analysis.  An  example  of 
these  factors  Is  provided  by  Salinger's  study  (1973),  which  found  Instances  of  Government 
clerical  workers  being  taught  writing  and  typing  styles  that  were  prohibited  on  the  job.  Another 
example  Is  that  Federal  requirements  concerning  equal  employment  opportunity  often  affect  the 
applicant  pool,  which  In  turn  affects  the  design  of  Instructional  programs. 


Research  Issues  Involving  Heeds  Assessment  and  Performance  Validity 

In  addition  to  the  research  Issues  concerning  needs  assessment  discussed  In  the  training 
validity  section,  the  organizational  analysis  issues  involved  In  transfer  suggest  a  number  of 
important  research  questions.  First,  it  Is  Important  to  identify  organizational  constraints  to 
transfer  performance.  Then,  It  Is  necessary  to  ask  how  that  should  affect  the  design  of 
Instructional  research  programs.  For  example,  It  is  clear  that  in  some  cases  supervisors  are  not 
ready  to  reward  the  behaviors  that  trainees  have  gained  as  a  result  of  Instruction,  nor  are  they 
willing  to  serve  as  role  models.  In  that  case,  one  question  Is  "How  does  that  affect  the 
maintenance  of  learned  behavior?"  Also,  It  Is  possible  to  build  instructional  systems  that 
compensate  for  that  problem.  All  of  these  kinds  of  questions  could  be  addressed  in  a  basic 
learning  laboratory. 

Evaluation  and  Performance  Validity 

A  reexamination  of  the  evaluation  section  for  training  validity  should  reinforce  the  point 
that  each  type  of  validity  being  considered  builds  on  the  concepts  presented  previously.  Thus, 


performance  validity  assumes  that  the  training  analyst  has  already  met  the  Issues  of  criterion 
development  and  evaluation  design  appropriate  for  the  establishment  of  training  validity* 
Clearly,  If  the  concern  Is  performance  validity,  there  Is  Increasing  complexity  In  the  design  of 
proper  evaluation  methodologies.  The  additional  difficulties  are  well  documented  by  Blalwes  and 
Regan's  (1970)  suggestion  that  evaluation  efforts  must  consider  three  criteria:  (a)  original 
learning  efficiency,  (b)  transfer  of  learning  to  the  new  task,  and  (c)  retention  of  original 
learning.  Yet,  In  the  area  of  flight  simulation,  where  the  greatest  effort  has  been  expended, 
the  emphasis  has  been  on  the  most  Innedlate  criterion,  original  learning;  and  even  those  studies 
have  been  plagued  by  serious  problems.  Basic  research  Involving  development  of  criteria  for  both 
training  and  performance  validity  Is  a  necessity. 

The  consideration  of  performance  validity  also  features  the  discussion  of  two  other 
evaluation  models  briefly  mentioned  In  the  training  validity  section.  They  are  Individual 
difference  models  and  content  validity  models. 


1.  Individual  Difference  Models 

Many  psychologists  have  emphasized  the  use  of  training  scores  as  a  way  to  predict  the  future 
success  of  potential  employees.  There  are  a  number  of  these  types  of  studies  showing  meaningful 
relationships  between  training  performance  and  on-the-job  performance.  For  example.  Kraut  (1975) 
found  that  peer  ratings  obtained  from  managers  attending  a  month-long  training  course  predicted 
several  criteria,  Including  future  promotion  and  performance  appraisal  ratings  of  job 
performance.  Other  Investigators  have  used  early  training  performance  to  predict  performance 
later  on  In  more  advanced  training.  An  example  of  this  approach  Is  offered  by  Gordon  and  Cohen 
(1973),  whose  study  Involved  a  welding  program  that  was  part  of  a  larger  manpower  development 
project.  The  program  consisted  of  14  different  tasks  that  fell  Into  four  categories  and  ranged 
In  difficulty  from  simple  to  complex.  Advancement  from  one  task  to  the  next  was  dependent  on 
successful  completion  of  all  previous  tasks.  Thus,  trainees  progressed  at  a  rate  commensurate 
with  their  ability  to  master  the  material  to  be  learned.  These  Investigators  understood  that 
they  were  predicting  the  performance  of  Individuals  on  a  later  task  (e.g.,  on  the  job  or  later  In 
training)  based  on  performance  In  the  training  program.  As  a  matter  of  fact,  once  these 
relationships  have  been  established  In  an  appropriately  designed  study.  It  Is  possible  to  select 
Individuals  for  a  job  or  for  later  training  based  on  these  training  scores.  In  other  words,  the 
training  score  serves  as  a  validated  predictor  of  future  performance.  However,  an  Important  note 
of  caution  should  be  considered  when  this  technique  Is  contemplated  as  an  evaluation  of  the 
training  program.  The  relationship  between  training  performance  and  on-the-job  performance 
simply  means  that  persons  who  perform  best  on  the  training  test  also  perform  well  on  the  job. 
This  does  not  necessarily  mean  that  the  training  program  Is  properly  designed  or  that  persons 
have  learned  enough  In  training  to  perform  well  on  the  job.  It  Is  entirely  possible  that  the 
training  program  did  not  teach  anything  or  that  the  trainees  did  not  learn  anything.  In  those 
cases,  there  could  still  be  Individual  differences  on  the  training  test.  Even  If  the  training 
program  did  not  teach  relevant  material  for  the  job,  there  would  still  be  a  strong  relationship 
If  the  person  who  did  well  on  the  test  did  well  on  the  job,  and  the  persons  who  did  poorly  on  the 
test  did  poorly  on  the  job.  In  other  words,  the  training  program  had  not  achieved  anything, 
except  to  maintain  the  Individual  differences  between  trainees  that  might  have  existed  before 
they  entered  tralrlng. 

2.  Content  Valid  Models 

If  the  needs  assessment  Is  appropriately  carried  out,  and  the  training  program  is  designed  to 
reflect  the  KSAs,  then  the  program  would  be  judged  as  having  content  validity.  In  other  words, 
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the  training  program  should  reflect  the  doaaln  of  KSAs  represented  on  the  job  that  the  analyst 
has  deteralned  should  be  learned  In  the  training  prograa.  One  way  of  conceptualizing  the  content 
validity  of  a  training  prograa  Is  presented  In  Figure  7-2.  In  this  figure,  the  horizontal  axis 
across  the  top  of  the  figure  represents  the  dlaenslon  of  the  laportance  or  criticality  of  the 
KSAs  as  deteralned  by  the  needs  assessment.  While  the  dlagraa  presents  KSAs  as  being  only 
laportant  or  not  laportant.  It  should  be  realized  that  this  Is  an  oversimplification  of  a 
dlaenslon  with  many  points.  The  vertical  dimension  represents  the  degree  of  emphasis  given  the 
KSAs  In  training.  Again,  to  simplify  the  presentation,  the  dimension  Is  presented  as  Indicating 
that  the  KSA  Is  or  Is  not  emphasized  In  the  training  prograa.  This  results  In  the  fourfold  table 
presented  In  Figure  7-2.  Using  this  approach,  both  boxes  A  and  0  provide  support  for  the  content 
validity  of  the  training  prograa.  KSAs  that  fall  Into  box  0  are  judged  as  being  Important  for 
the  job  and  should  be  emphasized  In  training.  Items  in  box  A  are  Judged  as  not  important  for  the 
job  and  should  not  be  emphasized  In  training.  Conceptually,  the  degree  to  which  KSAs  fall  into 
categories  A  and  D  make  It  possible  to  think  about  the  training  prograa  as  being  content  valid. 
Of  course,  this  Is  an  oversimplification.  There  will  be  KSAs  that  are  judged  as  moderately 
laportant  for  the  job  or  KSAs  that  are  moderately  emphasized  In  training.  However,  It  Is 
possible  to  conceive  of  this  type  of  relationship  and  actually  measure  the  degree  to  which  those 
KSAs  judged  as  Important  are  emphasized  (and  hopefully  learned)  In  training.  Very  few  persons 
would  be  unhappy  with  a  training  program  that  has  a  tendency  to  emphasize  the  objectives 
associated  with  KSAs  that  are  judged  critical  or  Important  for  job  performance. 


IMPORTANCE  OF  KSA 

(AS  DETERMINED  BY  NEEDS  ASSESSMENT) 


-NOT  IMPORTANT 
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Figure  7-2.  A  Conceptual  Diagram  of  Content  Validity  of  Training  Programs. 


Box  B  represents  a  potential  error  and  could  affect  the  degree  to  which  a  program  Is  judged 
content  valid.  KSAs  falling  Into  the  B-category  are  judged  as  Important  for  the  job  but  are  not 
emphasized  In  training.  From  a  systems  perspective,  these  Items  must  be  analyzed  to  determine 
whether  the  organization  Intends  for  these  KSAs  to  be  gained  as  a  result  of  training.  If  that  Is 
the  case,  then  there  Is  a  problem.  However,  It  Is  also  possible  that  Individuals  are  expected  to 
be  selected  with  that  particular  KSA  or  that  the  Individual  Is  expected  to  learn  the  material 
related  to  that  KSA  on  the  job.  To  that  extent,  the  training  program  should  not  be  expected  to 
emphasize  the  Item,  and  Its  content  validity  would  not  be  questioned.  However,  It  would  still  be 
Important  to  determine  that  KSAs  judged  as  Important  or  critical  are  covered  In  another  system 
such  as  selection.  If  the  Item  Is  not  covered,  then  the  organization  must  decide  whether 
revision  of  the  training  program  Is  necessary  or  whether  some  other  system  has  to  be  redesigned 
to  cover  that  material. 

Box  C  represents  KSAs  that  are  emphasized  In  training  but  are  judged  as  not  being  Important 
for  the  job.  This  Is  often  a  criticism  of  training  programs.  This  Is,  they  tend  to  spend  a  lot 
of  time  emphasizing  material  that  Is  not  job  related.  Most  analysts  agree  that  the  use  of  needs 
assessment  procedures  results  In  a  decrease  In  the  amount  of  training  time  necessary  to  complete 
the  program.  This  usually  comes  about  because  of  a  reduction  In  the  scope  of  the  program,  based 
upon  an  elimination  of  the  type  of  Items  present  in  category  C.  Interestingly,  a  systems  view  of 
this  process  might  even  suggest  a  reduction  of  Items  that  would  appear  in  category  D.  Selection 
experts  might  suggest  that  sometimes  KSAs  Included  In  category  D  are  unnecessary  because  they 
have  already  been  used  as  a  basis  for  selection.  Thus,  trainees  are  again  subjected  to  training 
materials  on  KSAs  that  are  already  in  their  repertoire. 


Research  Issues  Concerning  Evaluation  and  Performance  Validity 

Besides  the  criterion  Issues  mentioned  earlier,  basic  research  concerning  evaluation  models 
Is  badly  needed.  There  Is  virtually  no  research  comparing  experimental  design.  Individual 
differences,  and  content  evaluation  models.  There  are  very  limited  data  available  relating  what 
type  of  Individual  difference  measure  best  predicts  performance  at  various  stages  of  learning  on 
different  tasks.  Also,  there  are  virtually  no  data  available  on  appropriate  measures  of  degree 
of  content  validity  nor  on  the  relationship  between  data  on  content  validity  and  other  evaluation 
model s. 


Intra-Organlzatlonal  Validity 

Intra-organizatlonal  validity  assumes  that  the  trainer  has  established  training  and 
performance  validity  and  Is  now  concerned  with  predicting  the  performance  of  a  new  group  of 
trainees.  Just  as  performance  validity  assumes  the  consideration  of  the  points  established  for 
training  validity,  intra-organizatlonal  validity  assumes  that  the  points  discussed  for  training 
and  performance  validity  have  been  established.  It  should  be  noted  that  in  a  previously 
expressed  view,  evaluation  Is  considered  an  information-gathering  process  that  provides  feedback 
about  the  multiple  objectives  of  most  training  programs.  Thus,  it  becomes  apparent  that 
evaluation  should  be  a  continual  process  which  provides  data  as  a  basis  for  revisions  of  the 
program.  Mew  data  should  be  collected  based  on  the  performance  of  a  new  group  of  trainees,  which 
provides  further  understanding  about  the  achievement  of  objectives  or  about  the  variables  that 
affect  the  achievement  of  objectives.  That  does  not  mean  that  each  effort  must  start  from  the 
very  beginning.  However,  it  should  be  possible  to  collect  further  new  information  about  the 
effects  of  revisions.  Also,  it  should  be  possible  to  collect  data  that  can  be  checked  against 
previously  collected  information  to  make  sure  that  the  instructional  program  is  having  the  same 
effect. 
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Research  Needs  Concerning  Intra-Organizatlonal  Validity 

Although  there  Is  a  United  amount  of  data  that  help  us  to  understand  some  aspects  of 
training  and  performance  validity,  there  are  virtually  no  data  on  Intra-organizatlonal  validity* 
There  Is  Information  that  jobs  change,  persons  entering  jobs  change,  and  training  programs 
change;  but  there  are  no  data  on  how  this  affects  the  performance  of  Individuals  who  enter 
training  programs.  Most  everyone  would  agree  that  It  is  dangerous  to  generalize  to  new  trainee 
populations  when  they  are  not  being  instructed  In  the  same  training  program.  However,  most 
training  analysts  fall  to  realize  how  different  established  training  programs  tend  to  be  from  the 
original  training  program  that  was  evaluated.  Some  of  these  difficulties  can  be  labeled  problems 
of  reactivity.  The  excitement  generated  by  observers,  experimental  equipment,  and  questions 

being  asked  often  results  In  changes  In  behavior  that  are  not  a  result  of  the  actual  training 
program.  To  the  extent  that  these  variables  are  a  source  of  the  training  result,  and  to  the 
extent  that  these  variables  are  not  present  the  next  time  training  Is  offered,  general Izatlon  is 
difficult.  There  Is  a  certain  aura  of  excitement  surrounding  new  Instructional  treatments  or 
training  programs  that  simply  disappears  over  time.  In  some  studies  (e.g.,  Rosenthal,  1966), 
researchers  have  specified  how  the  expectations  of  the  experimenters  (or  trainers  or  teachers) 
themselves  have  an  effect  on  the  performance  of  Individuals.  To  the  extent  that  these  factors 
change  over  time,  the  training  program  has  changed,  and  It  is  difficult  to  generalize.  One  of 
the  most  compelling  reasons  for  the  use  of  process  criteria  Is  to  attempt  to  specify  these 

variables  and  their  effects.  When  the  process  criteria  Indicate  that  the  training  programs  have 
not  changed,  then  It  Is  obviously  safer  to  generalize.  A  training  laboratory  that  uses  programs 

over  long  periods  of  time,  with  different  groups  of  trainees,  has  a  unique  opportunity  to  study 

these  variables.  Researchers  might  even  be  able  to  gather  data  on  how  programs  tend  to  change. 
This  author  will  never  forget  the  look  of  astonishment  on  the  faces  of  a  number  of  high-level 
executives  who  had  just  discovered  that  the  reason  grocery  clerks  could  not  operate  the  cash 
register  was  because  this  Instructional  sequence  was  no  longer  part  of  the  training  program. 


Inter-Organizational  Validity 


In  this  Instance,  the  analyst  Is  attempting  to  determine  whether  a  training  program  validated 
in  one  organization  can  be  utilized  In  another  organization.  All  of  the  factors  discussed  in 
training  validity,  performance  validity,  and  Intra-organizatlonal  validity  affect  this  decision. 
As  indicated  In  the  section  on  Intra-organizatlonal  validity,  when  the  needs  assessment  shows 
differences  (l.e.,  the  task,  person,  or  organizational  components)  or  the  evaluation  is 
questionable  or  the  training  program  has  changed,  then  generalization  is  dangerous.  In  this 
Instance,  the  needs  assessment  and  evaluation  have  not  been  performed  for  the  organization  that 
desires  to  use  the  training  program.  Considering  the  vast  number  of  ways  organizations  differ. 
It  Is  dangerous  to  attempt  to  generalize  the  training  results  In  one  organization  to  trainees  In 
another  organization. 


Research  Meeds  Concerning  Inter-Organizational  Validity 

One  reason  it  Is  dangerous  to  generalize  is  that  we  know  so  little  about  the  effects  of 
variables  such  as  organizational  differences  on  the  design,  conduct  and  evaluation  of  training 
programs.  On  the  other  hand.  It  Is  entirely  appropriate  to  borrow  needs  assessment 

methodologies,  evaluation  strategies,  and  training  techniques  to  try  out  in  a  research 

laboratory.  Through  these  procedures,  organizations  can  establish  techniques  that  work,  and 

perhaps  It  will  be  possible  to  begin  to  understand  what  variables  affect  the  success  of  programs 

across  organizations. 
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SunMry  of  Research  Issues 


1.  Needs  Assessment  Issues 


Training  analysts  remain  uncertain  about  what  the  appropriate  Input  should  be  for  the  design 
of  Instructional  programs.  Needs  assessment  of  jobs  has  focused  on  a  number  of  different  factors 
Including  the  determination  of  the  tasks  to  be  performed,  the  KSAs  necessary  to  perform  the 
tasks,  and  the  behaviors  necessary  to  demonstrate  competent  performance.  However,  there  Is 
virtually  no  Information  available  comparing  the  various  systems  as  Inputs  to  the  design  of 
training  systems.  There  Is  not  much  Information  on  the  best  systems  to  determine  which 
characteristics  (e.g.,  frequency,  criticality)  should  be  measured  to  provide  Information  about: 
tasks,  KSAs,  or  performance.  A  consideration  of  the  kinds  of  parameters  that  are  directly 
relevant  to  the  Air  Force  might  suggest  different  types  of  strategies  for  various  KSAs.  For 
example.  It  may  be  possible  to  distinguish  between  skills  and  abilities  that  are  frequently 
performed  as  compared  to  those  skills  and  abilities  that  are  critical  to  performance  but  are 
rarely  performed.  A  related  Issue  is  that  the  Comprehensive  Occupational  Data  Analysis  Programs 
(COOAP)  task  analysis  system  currently  employed  by  the  Air  Force  includes  task-difficulty  ratings 
that  are  obtained  by  asking  supervisors  "How  long  does  It  take  to  learn  the  task?"  A  basic 
learning  laboratory  could  obtain  Important  validation  Information  by  comparing  the  measure  with 
actual  learning  time  on  the  task.  A  direct  measure  of  the  time  It  takes  to  learn  a  job  component 
would  make  possible  interesting  and  valuable  comparisons  with  task  difficulty  as  currently 
assessed.  The  opportunity  to  perform  needs  assessment  on  relevant  jobs  and  to  examine  the 
effects  of  the  utilized  system  on  Instructional  design  and  learner  performance  Is  a  unique 
opportunity  for  a  basic  learning  laboratory. 

In  addition,  organizational  analysis  Issues  Involved  In  transfer  suggest  a  number  of 
Important  research  questions.  First,  It  Is  Important  to  Identify  organizational  constraints  to 
transfer  performance.  Then,  It  Is  necessary  to  ask  how  that  should  affect  the  design  of 
Instructional  research  programs.  For  example,  one  commonly  heard  complaint  is  that  supervisors 
are  not  willing  to  serve  as  role  models  nor  reward  the  behavior  that  trainees  have  learned.  It 
Is  fairly  obvious  that  behavior  learned  In  basic  training,  but  not  supported  on  the  job,  may  not 
remain  In  the  learner's  repertoire.  However,  it  Is  likely  that  this  situation  occurs  In  many 
organizations.  Including  the  Air  Force.  A  basic  research  laboratory  would  provide  a  vehicle  for 
both  the  Identification  of  those  constraints  which  affect  transfer  and  an  opportunity  to  study 
the  effects.  For  example,  it  Is  quite  possible  to  study  the  learning  of  Inter-personal  behavior 
on  one  task  and  then  the  transfer  of  that  behavior  to  another  situation  In  the  laboratory.  In 
the  second  situation.  It  Is  possible  to  examine  situations  where  the  supervisor  Is  not  prepared 
to  support  or  reward  the  behavior.  It  Is  Important  to  understand  the  transfer  effects  in  those 
situations  and  also  to  explore  variables  In  the  learning  situation  to  determine  what  support 
variables.  If  any,  could  maintain  the  appropriate  behavior.  Another  organizational  constraint 
which  affects  transfer  behavior  Is  the  lack  of  opportunity  to  practice  the  learned  behavior  once 
the  person  Is  transferred  to  the  job.  A  study  by  Laabs,  Panell,  and  Pickering  (1977)  developed 
performance-oriented  tests  to  diagnose  deficiencies  in  job  performance  for  United  States  Navy 
missile  technicians.  This  study  found  that  performance  deficiencies  in  maintenance  skills  were 
related  to  the  fact  that  the  technicians  were  unable  to  practice  their  skills  because  of  the 
extreme  reliability  of  the  missile  test  and  readiness  equipment.  If  similar  situations  exist  in 
the  Air  Force,  the  organizational  analysis  would  suggest  the  importance  of  studying  procedures  in 
the  basic  learning  laboratory  that  would  support  behaviors  that  might  have  to  be  maintained 
without  actual  practice  over  long  periods  of  time. 
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2.  Research  Issues  Concerning  Criteria  for  Training  Validity 

In  general,  examination  of  the  types  of  criteria  employed  to  provide  evaluation  data  Is  not 
reassuring.  As  discussed  In  the  paper,  few  studies  have  bothered  to  measure  learning,  and 
on-the-job  behavior.  It  Is  probably  not  unreasonable  to  suspect  that  these  Investigations  did 
not  consider  criterion  relevance  and  reliability.  The  question  about  how  criterion  relevance  Is 
determined  Is  not  easily  answered.  There  should  be  a  scheme  for  the  tracing-out  process  from  the 
needs  assessment  to  the  development  of  criteria.  The  rules  of  this  process  are  unknown.  Indeed, 
Investigators  are  not  really  very  sure  what  the  Input  should  be  to  maximize  criterion  relevance. 
Recently  there  has  been  Increased  attention  to  criterion-referenced  measures  that  are  dependent 
on  an  absolute  standard  of  quality  (Swezey,  1978),  rather  than  norm-referenced  measures  that  are 
dependent  on  a  relative  standard.  The  characteristics  of  criterion-referenced  measures  - 
Including  methods  of  content  validity  to  determine  relevance  as  well  as  the  design  of  methods  to 
determine  reliability  Indices  -  need  attention. 

There  Is  even  great  difficulty  In  determining  what  constitutes  Ideal  performance  and  how  It 
should  be  measured.  As  discussed  above,  Wllllges,  Roscoe,  and  Wllllges  (1972)  note  that  there  Is 
little  agreement  on  what  constitutes  Ideal  pilot  performance,  and  the  reliability  of  most 
pilot-performance  grading  systems  Is  disappointing.  Thus,  even  studies  of  original  learning  are 
difficult  to  complete,  because  there  is  little  agreement  on  what  constitutes  terminal 
performance.  Even  In  the  area  of  simulators,  where  there  have  been  extensive  efforts,  the 
establishment  of  appropriate  criteria  for  complex  performance  Is  a  serious  Issue  demanding  the 
attention  of  researchers.  In  addition,  continuing  questions  concern  the  development  of  new 
criteria  measures  that  will  provide  further  information  about  the  Instructional  process.  Thus, 
reaction  measures  that  only  ask  the  trainee  for  general  feelings  about  learning  have  not  been 
judged  particularly  useful.  However,  It  Is  possible  that  trainees  are  able  to  self-assess  their 
learning  of  particular  components  of  KSAs.  Research  Is  needed  to  determine  whether  learners  can 
assess  their  own  capabilities  concerning  how  much  they  have  learned.  There  are  also  questions 
concerning  the  relationships  between  these  self-assessment  Indices  and  other  measures  of  learning 
In  the  Instructional  environment.  A  basic  learning  laboratory  would  provide  an  important  vehicle 
to  explore  these  Issues. 


Conclusions 

This  paper  has  presented  the  author's  views  concerning  a  hierarchy  in  the  establishment  of 
training,  performance,  and  Intra-organlzatlonal  and  Inter-organizational  validity.  One  way  for 
the  training  analyst  to  consider  this  entire  process  Is  to  ask  "What  Is  the  final  goal?"  If  the 
final  goal  Is  the  establishment  of  Intra-organlzatlonal  validity,  then  it  Is  necessary  to 
establish  all  of  the  components  of  validity  that  precede  It  (I.e.,  both  training  and  performance 
validity). 

The  examination  of  the  needs  assessment  processes  and  evaluation  processes  as  related  to  each 
of  the  validity  goals  raises  a  number  of  interesting  Instructional  systems  Issues  that  should  be 
examined  in  a  learning  laboratory. 
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CHAPTER  8 


RESEARCH  ISSUES 

Irwin  L.  Goldstein 
University  of  Maryland 

This  paper  briefly  summarizes  the  research  Issues  presented  In  the  preceding  chapters.  The 
authors  have  presented  their  own  Individual  views  concerning  contemporary  research  and  theory,  as 
related  to  the  formation  of  a  research  agenda  for  a  learning  research  laboratory.  Interestingly, 
these  Individual  chapters  result  In  many  common  themes.  The  purpose  of  this  chapter  Is  to 
summarize  and  describe  those  themes. 


Acquisition  and  Retention  Issues 

Glaser's  chapter  focuses  on  the  principle  that  a  major  objective  of  Air  Force  training  should 
be  to  produce  expert  apprentices.  He  defines  expert  apprentices  as  people  who,  upon  leaving 
training,  have  strong  capabilities  to  profit  from  the  experiences  they  will  receive  in  their 
military  careers.  Many  of  these  Issues  refer  to  transfer-of-tralnlng  questions.  What  does  the 
learner  have  to  acquire  in  training  to  result  fn  more  effective  learning  on  the  job?  However, 
Glaser  then  takes  the  reader  back  a  step  by  saying  that  first  It  is  necessary  to  understand  what 
most  persons  must  acquire  In  order  to  be  ready  to  profit  from  their  experiences.  Glaser  then 
provides  a  number  of  parameters  that  should  be  studied  In  a  learning  laboratory.  Some  of  these 
are  variables  such  as  categorization  ability,  knowledge  of  principles,  mental  models,  and 
automatization.  Glaser  further  asks  questions  such  as:  How  Is  knowledge  categorized  and  what 
principles  must  be  gained  in  addition  to  understanding  the  procedures  required?  In  addition, 
there  are  Issues  about  what  conditions  of  Instruction  favor  the  development  of  automatlclty  In 
basic  skills. 

Gagnl  addresses  Issues  similar  to  those  of  Glaser  and  asks  how  It  Is  possible  to  continue  to 
maintain  skills  that  are  not  constantly  being  used.  He  notes  that  these  Issues  suggest  a  whole 
set  of  research  questions  about  the  adequacy  of  learning  and  their  effects  on  later  performance. 
Gagn£  suggests  that  It  Is  necessary  to  begin  to  understand  how  learning  proceeds  from  Initial 
acquisition  of  a  procedure  to  later  stages  of  adequacy.  He  also  notes  the  need  to  understand 
composition,  which  refers  to  how  certain  parts  of  a  procedure  are  consolidated  and  also 
automatized  so  that  its  performance  does  not  Interfere  with  another  task  concurrently  being 
performed.  Gagnl  not  only  emphasizes  research  on  variables  affecting  skill  retention  but  also 
brings  attention  to  the  point  that  when  training  courses  are  of  lengthy  duration,  skills  learned 
early  may  be  forgotten  by  the  end  of  the  program.  An  examination  of  this  type  of  training 
program  might  provide  a  perfect  vehicle  for  a  learning  laboratory  to  study  skill  retention 
Issues.  Gagn£  also  emphasizes  research  on  relearning,  noting  that  one  way  of  treating  the  skill 
retention  problem  is  to  emphasize  relearning  procedures. 

In  his  chapter,  Goldstein  notes  that  there  are  a  number  of  situations  where  the  extremely 
high  reliability  of  the  technical  equipment  results  in  persons  not  having  the  opportunity  to 
perform  a  task  except  In  emergencies  when  the  equipment  falls.  He  calls  for  the  study  of 
procedures  which  would  support  behaviors  that  might  have  to  be  maintained  without  actual  practice 
over  a  long  period  of  time.  Goldstein  adds  that  there  has  been  little  attention  given  to 
studying  variables  In  the  job  situation  which  support  learning  that  has  taken  place  in  training. 
One  variable  that  he  suggests  should  be  examined  is  the  effects  of  maintenance  of  learned 
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behaviors  In  situations  where  the  supervisor  Is  or  Is  not  willing  to  serve  as  role  model  or 
reward  the  learned  behavior. 

Royer  also  focuses  on  similar  issues.  His  paper  discusses  techniques  that  either  activate 
relevant  prior  knowledge  or  establish  a  bridge  between  prior  knowledge  and  material  to  be 
learned.  For  example,  Royer  suggests  that  mnemonic  techniques  and  the  use  of  analogies  for 
Instructional  purposes  appear  to  hold  a  great  deal  of  promise.  He  also  notes  that  while  some  of 
these  suggestions  (e.g.,  mnemonic  strategies)  have  been  examined  in  educational  settings,  there 
is  virtually  no  research  on  the  utility  of  these  techniques  in  military  training  situations. 
Royer  also  notes  that  some  techniques,  such  as  the  use  of  analogies  in  instructional  settings, 
are  just  beginning  to  be  researched  and  would  require  intensive  investigations  In  a  learning 
laboratory. 

Shuell  points  out  that  most  learning  for  persons  in  the  Air  Force  Is  procedural  in  nature,  as 
in  repairing  a  piece  of  equipment.  However,  he  notes  that  declarative  knowledge  such  as  learning 
a  radio  call  sign  Is  also  Important.  Yet,  there  is  not  much  information  about  the  relationship 
between  these  two  forms  of  learning.  Shuell  asks  whether  emphasis  on  the  declarative  knowledge 
early  In  the  learning  process  might  be  helpful  In  learning  the  procedural  tasks  required  in  Air 
Force  jobs.  In  examining  this  issue,  Shuell  also  notes  the  Importance  of  examining  a  number  of 
Instructional  variables,  such  as  visual  mnemonics  and  behavioral  objectives. 


Individual  Difference  Issues 

Payne  noted  the  need  to  determine  the  relationship  between  aptitude  and  training  time.  The 
Issues  that  he  raises  are  captured  by  asking  what  the  payoff  Is  in  training  In  recruiting 
individuals  with  Increased  aptitude  scores.  In  addition,  he  notes  the  Importance  of  learning  how 
these  relationships  are  affected  by  the  nature  of  the  skill  being  learned  and  the  aptitude  chosen 
for  examination. 

Shuell  also  emphasizes  this  question,  pointing  out  that  such  concerns  must  emphasize  the 
Individualization  of  instruction  as  well  as  questions  about  individual  differences  of  people. 
Shuell  asks  a  whole  set  of  interesting  questions  including  "How  do  experts  who  perform  Air  Force 
jobs  differ  from  trainees?"  He  suggests  that  by  understanding  such  differences,  it  will  be 
possible  to  gain  insight  to  Instructional  interventions  that  will  be  important  to  the  learning 
process.  Similarly  Shuell  would  also  ask  what  differentiates  the  performance  of  "good"  and 
"poor"  learners  at  various  levels  of  competence. 

Goldstein  also  discussed  the  Importance  of  individual  differences  research.  He  notes  that 
there  is  very  little  Information  available  to  Indicate  what  type  of  individual  difference  measure 
best  predicts  performance  at  various  stages  of  learning  on  different  tasks.  This  type  of 
information  could  be  very  useful  in  determining  when  individuals  in  a  training  program  should  be 
moved  on  to  the  next  learning  task. 

Gagn£  also  discusses  the  use  of  Individual  difference  methodology  to  validate  aptitude  test 
scores  against  job-component  tasks  designed  for  research  in  the  learning  laboratory.  Thus,  by 
investigating  learning  performance  on  relevant  tasks,  the  laboratory  also  affords  the  opportunity 
to  examine  the  predictability  of  various  aptitude  measures. 
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Meeds  Assessment  Issues 


Goldstein  notes  that  training  analysts  remain  uncertain  about  the  appropriate  Input  for  the 
design  of  Instructional  programs.  He  Indicates  that  needs  assessment  of  jobs  has  focused  on  a 
number  of  different  factors,  Including  the  determination  of  the  tasks  to  be  performed;  the 
knowledge,  skills,  and  abilities  necessary  to  perform  the  tasks;  and  the  behaviors  necessary  to 
demonstrate  competent  performance.  However,  there  Is  little  Information  available  comparing  the 
various  systems  as  Inputs,  nor  Is  there  much  Information  about  the  utility  of  descriptive 
characteristics  (e.g.,  frequency,  criticality)  to  provide  Information  about:  (a)  tasks,  (b) 
knowledge,  (c)  skills  or  abilities,  or  (d)  performance.  For  example,  a  consideration  of  the 
kinds  of  parameters  that  are  directly  relevant  to  the  Air  Force  might  suggest  different  types  of 
training  strategies  for  skills  and  abilities  that  are  frequently  performed,  as  compared  to  those 
skills  and  abilities  which  are  critical  to  performance  but  are  rarely  performed.  Both  Goldstein 
and  Gagnd  stress  the  same  Issue  when  they  note  that  the  CODAP  database  Includes  ratings  of  task 
difficulty  that  are  obtained  by  asking  supervisors  “How  long  does  It  take  to  learn  the  task?" 
They  Indicate  that  the  measure  needs  to  be  validated  by  comparing  It  with  direct  measures  of 
"time  to  learn"  on  the  same  task.  As  they  both  Indicate,  this  would  make  possible  some 
Interesting  comparisons  with  the  task  difficulty  database. 

Gagnd  also  points  out  that  there  are  many  different  approaches  to  the  selection  of  job  tasks 
which  are  representative.  Research  examining  the  representativeness  of  Job  components,  using 
both  the  CODAP  system  and  measures  such  as  time  to  learn,  would  provide  Important  Information  on 
how  to  establish  procedures  to  select  representative  job  components.  Royer  In  his  ISD  model  also 
emphasizes  the  needs  assessment  process.  His  model  begins  with  transfer  task  analysis.  The  Idea 
here  Is  to  analyze  the  domain  of  activities  a  trainee  will  engage  In  after  training  Is 
completed.  Royer  Indicates  the  activities  can  then  be  used  to  serve  as  Indices  of  the  ability  of 
training  to  provide  transfer.  This  type  of  model,  based  on  needs  assessment  procedures  to  select 
transfer  objectives,  could  result  In  the  design  of  learning  tasks  that  are  much  more  likely  to 
produce  transfer  onto  the  job. 


Criterion  and  Measurement  Issues 

All  of  the  papers  stress  the  Importance  of  research  on  criterion  Issues.  Gagnd  notes  that 
low  reliability  has  often  been  an  obstacle  to  the  use  of  job  performance  measures.  He  suggests 
the  need  for  further  research  on  criterion-reference  measurement  and  the  development  of 
appropriate  measures  of  reliability.  Goldstein  makes  the  same  point  and  also  suggests  that  the 
development  of  content  validity  procedures  to  validate  criteria  Is  badly  needed.  He  notes  that 
there  Is  great  difficulty  In  determining  what  constitutes  ideal  pilot  performance  and  how  such 
performance  should  be  measured.  Even  studies  of  learning  are  difficult  to  complete  because  there 
Is  little  agreement  on  what  constitutes  terminal  performance.  Royer  further  attests  to  the  need 
for  this  research  when  he  Indicates  that  assessment  of  knowledge  specifically  relevant  to  a 
particular  course  should  be  tested  as  a  predictor  of  end-of-tralning  performance.  He  then 
Indicates  that  the  key  to  evaluating  this  hypothesis  Is  to  devise  measurement  procedures  that 
reflect  performance  of  a  relevant  activity  actually  Included  in  training.  What  types  of  measure 
will  be  most  useful  remains  a  question.  Also,  whether  different  types  of  measures  will  best 
predict  performance  at  different  points  In  training  also  remains  a  research  question.  As  Gagn£ 
notes,  widespread  evaluation  of  job  performance  on  site  In  the  Air  Force  would  be  prohibitively 
expensive.  Validation  of  both  learning  and  job  performance  measures  would  be  an  important  task 
for  a  learning  laboratory. 

The  topics  above  present  only  a  few  of  the  research  themes  mentioned  in  many  of  the  papers. 
It  should  be  noted  that  there  were  many  other  Interesting  and  important  topics  discussed  In  the 
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Individual  papers.  For  example,  Glaser  presents  some  Important  research  suggestions  about 
teaching  and  management  variables  that  should  be  Investigated.  Thus,  the  Individual  papers 
should  be  examined  In  order  to  gain  an  appreciation  for  the  large  number  of  Important  topics  that 
could  be  researched  In  a  learning  laboratory.  Also,  of  course,  the  suoury  statements  contained 
In  this  paper  do  not  do  justice  to  the  breadth  of  Ideas  expressed  In  the  papers.  Again,  the 
papers  oust  be  read  Individually  to  accomplish  that. 


CONCLUDING  REMARKS 

Lt  Col  David  L.  Payne 
Air  Force  Human  Resources  Laboratory 

Dr.  Goldstein  has  presented  an  excellent  simmary  In  this  chapter  of  the  major  themes  of  the 
report  contributors.  As  he  noted,  the  research  proposals  are  numerous  and  have  merit  -  both  In 
their  contributions  to  the  general  body  of  knowledge  In  the  leamlng/tralnlng  research  areas,  and 
In  their  potential  utility  for  Improving  current  military  training  programs.  I  would  like  to 
conclude  by  providing  a  brief  statement  as  to  my  opinion  regarding  ‘Given  all  this  -  where  do  we 
go  from  here?*  In  doing  so,  I  provide  the  perspective  of  a  research  psychologist  who  Is  familiar 
with  the  various  military  training  systems  and  requirements. 

First,  the  studies  proposed  that  are  natural  follows-ons  to  research  now  being  conducted  as 
part  of  the  AFWL  Learning  Abilities  Measurement  Program  (Project  LAMP)  should  be  given  high 
priority.  As  the  Project  LAMP  results  help  us  to  Identify  the  components  of  learning  ability, 
the  next  step  Is  to  determine  how  these  components  are  modified  by  training  and  what 
Instructional  conditions  are  best  suited  to  the  task.  As  an  example,  what  Instructional 

techniques  enhance  the  learning  capabilities  of  those  who  are  deficient  In  working-memory 
capacity?  Or,  as  noted  by  Dr.  Glaser,  what  conditions  favor  automatlclty  In  the  acquisition  of 
basic  skills?  These  are  Just  two  of  the  numerous  research  efforts  which  could  capitalize  on  the 
results  of  on-going  LAMP  studies  focusing  on  Individual  differences  In  Information-processing 
skills. 

Second,  as  suggested  by  Dr.  Gagnd,  further  research  should  be  undertaken  concerning  the 
retention  of  learned  skills.  Results  from  earlier  Project  LAMP  studies  Indicate  that  the 
entering  Air  Force  recruits  vary  widely  In  their  learning  abilities,  and  their  retention  appears 
to  be  highly  related  to  Initial  learning  rate.  Some  questions  that  should  be  addressed  In  this 
area  Include  the  following:  What  are  the  effects  of  skill  acquisition  rates  on  the  retention  of 
different  types  of  Information?  What  techniques  can  be  used  on  the  job  to  enhance/maintain  the 
retention  of  learned  skills  (Goldstein)?  What  are  the  effects  of  different  Instructional 

techniques  on  retention  (e.g. ,  mnemonics)?  What  techniques  are  best  suited  for  the  retention  of 
different  types  of  knowledge  (l.e.,  procedural  versus  declarative)?  This  stream  of  research 

could  have  great  utility  for  the  military  services  In  determining  the  appropriate  lengths  and 

Instructional  modalities  of  various  courses  In  consideration  of  desired  student  retention  and 
relearning  requirements. 

Finally,  several  authors  noted  the  critical  need  for  further  research  on  criterion  measures. 
The  military  services'  personnel  research  laboratories  are  now  participating  In  a  joint  effort  to 
develop  reliable  measures  of  job  performance.  Their  first  efforts  will  be  aimed  at  developing 
surrogates  of  actual  "hands-on"  performance  measures.  The  surrogate  measures  can  then  be  used  to 
validate  the  services'  selection  and  training  procedures  In  an  economically  feasible  fashion. 
The  products  of  their  efforts  could  also  provide  end-of -course  criterion  measures  that  could  be 
tested  as  alternatives  to  current  measures.  The  following  two  questions  could  be  addressed:  What 
are  the  effects  of  using  different  Instructional  techniques  on  student  performance  using  the 
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different  criterion  aeasures?  Is  the  same  type  of  learning  Involved  when  using  the  different 
measures?  An  exaaple  for  the  latter  question  Is:  Learning  required  to  pass  a  job  performance 
test  may  be  more  procedural  In  nature  than  learning  required  to  pass  a  written  end-of-block  test 
(declarative). 

In  conclusion,  I  would  like  to  express  *y  sincere  appreciation  to  the  authors  of  these 
papers.  Their  truly  outstanding  efforts  have  provided  the  foundation  for  at  least  a  decade  of 
valuable  research  In  the  learning  and  training  areas. 


